DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
@ 2020-09-16 12:00 Dey, Souvik
  2020-09-16 14:18 ` Kevin Traynor
  0 siblings, 1 reply; 12+ messages in thread
From: Dey, Souvik @ 2020-09-16 12:00 UTC (permalink / raw)
  To: dev; +Cc: Stephen Hemminger, longli, Yigit, Ferruh, sthemmin

Hi All,
      I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/ . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we are all good but any larger packets/fragmented packets are getting dropped after some time. As soon as we start to receive the transmit completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why this is happening and is there is some fix which is already there which is done after 18.11.9 release ?
We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.

PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/>  fix in but nothing changed.

--
Regards,
Souvik

Log Snippet:
I can see the below errors coming after some transmitting.
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_flush_txagg() tx: port 2:0 tx 2048 size 302
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242

hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_flush_txagg() tx: port 2:0 tx 2624 size 302
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242

hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_flush_txagg() tx: port 2:0 tx 2688 size 302
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
hn_flush_txagg() tx: port 2:0 tx 3264 size 571
hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
hn_flush_txagg() tx: port 2:0 tx 3328 size 571
hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
hn_flush_txagg() tx: port 2:0 tx 3392 size 120
hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-16 12:00 [dpdk-dev] Issue with net/netvsc pmd in 18.11.9 Dey, Souvik
@ 2020-09-16 14:18 ` Kevin Traynor
  2020-09-16 14:41   ` Dey, Souvik
  0 siblings, 1 reply; 12+ messages in thread
From: Kevin Traynor @ 2020-09-16 14:18 UTC (permalink / raw)
  To: Dey, Souvik, dev; +Cc: Stephen Hemminger, longli, Yigit, Ferruh, sthemmin

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
>       I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/ . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we 

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
> 
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/>  fix in but nothing changed.
> 
> --
> Regards,
> Souvik
> 
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
> 
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
> 
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
> 


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-16 14:18 ` Kevin Traynor
@ 2020-09-16 14:41   ` Dey, Souvik
  2020-09-16 17:43     ` Long Li
  0 siblings, 1 reply; 12+ messages in thread
From: Dey, Souvik @ 2020-09-16 14:41 UTC (permalink / raw)
  To: Kevin Traynor, dev; +Cc: Stephen Hemminger, longli, Yigit, Ferruh, sthemmin

Yes the patch solves the seg fault issue, but even after backporting the patch we are not able to send packets with size higher than 512. The packets not being transmitted is the real problem here. I have also tried to take the 18.11.10rc1 build but faced similar issues in transmitting the packets.

--
Regards,
Souvik

From: dev <dev-bounces@dpdk.org> On Behalf Of Kevin Traynor
Sent: Wednesday, September 16, 2020 10:18 AM
To: Dey, Souvik <sodey@rbbn.com>; dev@dpdk.org
Cc: Stephen Hemminger <stephen@networkplumber.org>; longli@microsoft.com; Yigit, Ferruh <ferruh.yigit@intel.com>; sthemmin@microsoft.com
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
> I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/<https://patches.dpdk.org/patch/75001> . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/<http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com>

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html<http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html>

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
>
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/<https://patches.dpdk.org/patch/75118>> fix in but nothing changed.
>
> --
> Regards,
> Souvik
>
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
>

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-16 14:41   ` Dey, Souvik
@ 2020-09-16 17:43     ` Long Li
  2020-09-16 19:43       ` Dey, Souvik
  0 siblings, 1 reply; 12+ messages in thread
From: Long Li @ 2020-09-16 17:43 UTC (permalink / raw)
  To: Dey, Souvik, Kevin Traynor, dev
  Cc: Stephen Hemminger, Yigit, Ferruh, Stephen Hemminger

Hi Souvik,

Please also pick up this patch:
https://patchwork.dpdk.org/patch/75341/

This patch didn’t make it to 18.11.

Let me know if you are still seeing failures. If you can reproduce the failure with testpmd or other tools in DPDK, please also share the command you used.

Thanks,
Long

From: Dey, Souvik <sodey@rbbn.com>
Sent: Wednesday, September 16, 2020 7:41 AM
To: Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org
Cc: Stephen Hemminger <stephen@networkplumber.org>; Long Li <longli@microsoft.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

Yes the patch solves the seg fault issue, but even after backporting the patch we are not able to send packets with size higher than 512. The packets not being transmitted is the real problem here. I have also tried to take the 18.11.10rc1 build but faced similar issues in transmitting the packets.

--
Regards,
Souvik

From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Kevin Traynor
Sent: Wednesday, September 16, 2020 10:18 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; longli@microsoft.com<mailto:longli@microsoft.com>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
> I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130122587&sdata=Y9OdJYjrfRA2uDr%2Bs%2FMyqmYFZjPwsXPWV%2BvYWVTSy04%3D&reserved=0> . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox.dpdk.org%2Fstable%2F20200901100115.72365-1-ktraynor%40redhat.com&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130132583&sdata=BkEMWpgwjXSKZaZTTr3XN1fGWzuwv3%2BEQQOxyzy0SEc%3D&reserved=0>

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides-18.11%2Frel_notes%2Frelease_18_11.html&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130132583&sdata=Nv6Fp%2BzkErUTsZbzHZbXi8NA7bDLAEPzIEnHLLejb6c%3D&reserved=0>

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
>
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75118&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130142578&sdata=aiz%2FzHqmsH%2F8WbKSpsCalybZPZqBXgw1FWJJuYnLfgE%3D&reserved=0>> fix in but nothing changed.
>
> --
> Regards,
> Souvik
>
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
>

________________________________
Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.
________________________________

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-16 17:43     ` Long Li
@ 2020-09-16 19:43       ` Dey, Souvik
  2020-09-16 20:32         ` Dey, Souvik
  0 siblings, 1 reply; 12+ messages in thread
From: Dey, Souvik @ 2020-09-16 19:43 UTC (permalink / raw)
  To: Long Li, Kevin Traynor, dev
  Cc: Stephen Hemminger, Yigit, Ferruh, Stephen Hemminger

Hi Long,

I did take this patch on top of 18.11.10rc1 , but with this patch and also with 18.11.10rc1 files, the rx itself is not working and dhcp fails. With 18.11.9 the rx is working fine and changes present in 18.11.10rc1(hn_rxtx.c  ) does not look to work. 18.11.6 was all proper but 18.11.9 is not helping. I am not using testpmd but using our own dpdk app on Azure.
We are using the below command line args:
-c 0xe -n 3 -w 83f0:00:02.0 -w 8cb0:00:02.0 --vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1 --log-level pmd.netvsc:debug --proc-type=primary

Thanks in advance for all the help.

--
Regards,
Souvik

From: Long Li <longli@microsoft.com>
Sent: Wednesday, September 16, 2020 1:44 PM
To: Dey, Souvik <sodey@rbbn.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

Hi Souvik,

Please also pick up this patch:
https://patchwork.dpdk.org/patch/75341/<https://patchwork.dpdk.org/patch/75341>

This patch didn’t make it to 18.11.

Let me know if you are still seeing failures. If you can reproduce the failure with testpmd or other tools in DPDK, please also share the command you used.

Thanks,
Long

From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 7:41 AM
To: Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

Yes the patch solves the seg fault issue, but even after backporting the patch we are not able to send packets with size higher than 512. The packets not being transmitted is the real problem here. I have also tried to take the 18.11.10rc1 build but faced similar issues in transmitting the packets.

--
Regards,
Souvik

From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Kevin Traynor
Sent: Wednesday, September 16, 2020 10:18 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; longli@microsoft.com<mailto:longli@microsoft.com>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
> I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130122587&sdata=Y9OdJYjrfRA2uDr%2Bs%2FMyqmYFZjPwsXPWV%2BvYWVTSy04%3D&reserved=0> . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox.dpdk.org%2Fstable%2F20200901100115.72365-1-ktraynor%40redhat.com&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130132583&sdata=BkEMWpgwjXSKZaZTTr3XN1fGWzuwv3%2BEQQOxyzy0SEc%3D&reserved=0>

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides-18.11%2Frel_notes%2Frelease_18_11.html&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130132583&sdata=Nv6Fp%2BzkErUTsZbzHZbXi8NA7bDLAEPzIEnHLLejb6c%3D&reserved=0>

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
>
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75118&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130142578&sdata=aiz%2FzHqmsH%2F8WbKSpsCalybZPZqBXgw1FWJJuYnLfgE%3D&reserved=0>> fix in but nothing changed.
>
> --
> Regards,
> Souvik
>
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
>

________________________________
Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.
________________________________

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-16 19:43       ` Dey, Souvik
@ 2020-09-16 20:32         ` Dey, Souvik
  2020-09-16 22:07           ` Dey, Souvik
  0 siblings, 1 reply; 12+ messages in thread
From: Dey, Souvik @ 2020-09-16 20:32 UTC (permalink / raw)
  To: Long Li, Kevin Traynor, dev
  Cc: Stephen Hemminger, Yigit, Ferruh, Stephen Hemminger

If it helps the tx_conf->tx_free_thresh is set to 0 from app side and nb_desc is set to 4096 from the app. The same was set in 18.11.6 too.

From: Dey, Souvik
Sent: Wednesday, September 16, 2020 3:44 PM
To: 'Long Li' <longli@microsoft.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

Hi Long,

I did take this patch on top of 18.11.10rc1 , but with this patch and also with 18.11.10rc1 files, the rx itself is not working and dhcp fails. With 18.11.9 the rx is working fine and changes present in 18.11.10rc1(hn_rxtx.c  ) does not look to work. 18.11.6 was all proper but 18.11.9 is not helping. I am not using testpmd but using our own dpdk app on Azure.
We are using the below command line args:
-c 0xe -n 3 -w 83f0:00:02.0 -w 8cb0:00:02.0 --vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1 --log-level pmd.netvsc:debug --proc-type=primary

Thanks in advance for all the help.

--
Regards,
Souvik

From: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>
Sent: Wednesday, September 16, 2020 1:44 PM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

Hi Souvik,

Please also pick up this patch:
https://patchwork.dpdk.org/patch/75341/<https://patchwork.dpdk.org/patch/75341>

This patch didn’t make it to 18.11.

Let me know if you are still seeing failures. If you can reproduce the failure with testpmd or other tools in DPDK, please also share the command you used.

Thanks,
Long

From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 7:41 AM
To: Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

Yes the patch solves the seg fault issue, but even after backporting the patch we are not able to send packets with size higher than 512. The packets not being transmitted is the real problem here. I have also tried to take the 18.11.10rc1 build but faced similar issues in transmitting the packets.

--
Regards,
Souvik

From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Kevin Traynor
Sent: Wednesday, September 16, 2020 10:18 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; longli@microsoft.com<mailto:longli@microsoft.com>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
> I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130122587&sdata=Y9OdJYjrfRA2uDr%2Bs%2FMyqmYFZjPwsXPWV%2BvYWVTSy04%3D&reserved=0> . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox.dpdk.org%2Fstable%2F20200901100115.72365-1-ktraynor%40redhat.com&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130132583&sdata=BkEMWpgwjXSKZaZTTr3XN1fGWzuwv3%2BEQQOxyzy0SEc%3D&reserved=0>

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides-18.11%2Frel_notes%2Frelease_18_11.html&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130132583&sdata=Nv6Fp%2BzkErUTsZbzHZbXi8NA7bDLAEPzIEnHLLejb6c%3D&reserved=0>

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
>
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75118&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130142578&sdata=aiz%2FzHqmsH%2F8WbKSpsCalybZPZqBXgw1FWJJuYnLfgE%3D&reserved=0>> fix in but nothing changed.
>
> --
> Regards,
> Souvik
>
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
>

________________________________
Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.
________________________________

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-16 20:32         ` Dey, Souvik
@ 2020-09-16 22:07           ` Dey, Souvik
  2020-09-17  6:57             ` Long Li
  0 siblings, 1 reply; 12+ messages in thread
From: Dey, Souvik @ 2020-09-16 22:07 UTC (permalink / raw)
  To: Long Li, Kevin Traynor, dev
  Cc: Stephen Hemminger, Yigit, Ferruh, Stephen Hemminger

Looks like the below patch has changes a lot in the tx path which came in 18.11.9. Verified till 18.11.8 everything is working as expected.

https://patches.dpdk.org/patch/67511/

Can you confirm if this change or the recv change patch proposed below will require any specific device config change as my same app does not work any longer.

--
Regards,
Souvik

From: Dey, Souvik
Sent: Wednesday, September 16, 2020 4:32 PM
To: Long Li <longli@microsoft.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

If it helps the tx_conf->tx_free_thresh is set to 0 from app side and nb_desc is set to 4096 from the app. The same was set in 18.11.6 too.

From: Dey, Souvik
Sent: Wednesday, September 16, 2020 3:44 PM
To: 'Long Li' <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

Hi Long,

I did take this patch on top of 18.11.10rc1 , but with this patch and also with 18.11.10rc1 files, the rx itself is not working and dhcp fails. With 18.11.9 the rx is working fine and changes present in 18.11.10rc1(hn_rxtx.c  ) does not look to work. 18.11.6 was all proper but 18.11.9 is not helping. I am not using testpmd but using our own dpdk app on Azure.
We are using the below command line args:
-c 0xe -n 3 -w 83f0:00:02.0 -w 8cb0:00:02.0 --vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1 --log-level pmd.netvsc:debug --proc-type=primary

Thanks in advance for all the help.

--
Regards,
Souvik

From: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>
Sent: Wednesday, September 16, 2020 1:44 PM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

Hi Souvik,

Please also pick up this patch:
https://patchwork.dpdk.org/patch/75341/<https://patchwork.dpdk.org/patch/75341>

This patch didn’t make it to 18.11.

Let me know if you are still seeing failures. If you can reproduce the failure with testpmd or other tools in DPDK, please also share the command you used.

Thanks,
Long

From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 7:41 AM
To: Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

Yes the patch solves the seg fault issue, but even after backporting the patch we are not able to send packets with size higher than 512. The packets not being transmitted is the real problem here. I have also tried to take the 18.11.10rc1 build but faced similar issues in transmitting the packets.

--
Regards,
Souvik

From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Kevin Traynor
Sent: Wednesday, September 16, 2020 10:18 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; longli@microsoft.com<mailto:longli@microsoft.com>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
> I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130122587&sdata=Y9OdJYjrfRA2uDr%2Bs%2FMyqmYFZjPwsXPWV%2BvYWVTSy04%3D&reserved=0> . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox.dpdk.org%2Fstable%2F20200901100115.72365-1-ktraynor%40redhat.com&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130132583&sdata=BkEMWpgwjXSKZaZTTr3XN1fGWzuwv3%2BEQQOxyzy0SEc%3D&reserved=0>

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides-18.11%2Frel_notes%2Frelease_18_11.html&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130132583&sdata=Nv6Fp%2BzkErUTsZbzHZbXi8NA7bDLAEPzIEnHLLejb6c%3D&reserved=0>

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
>
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75118&data=02%7C01%7Clongli%40microsoft.com%7C1f1640e423394eb94f6e08d85a4ee081%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358642130142578&sdata=aiz%2FzHqmsH%2F8WbKSpsCalybZPZqBXgw1FWJJuYnLfgE%3D&reserved=0>> fix in but nothing changed.
>
> --
> Regards,
> Souvik
>
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
>

________________________________
Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.
________________________________

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-16 22:07           ` Dey, Souvik
@ 2020-09-17  6:57             ` Long Li
  2020-09-17 14:54               ` Dey, Souvik
  0 siblings, 1 reply; 12+ messages in thread
From: Long Li @ 2020-09-17  6:57 UTC (permalink / raw)
  To: Dey, Souvik, Kevin Traynor, dev
  Cc: Stephen Hemminger, Yigit, Ferruh, Stephen Hemminger

The patch below is correct. I'm wondering if you have missed another patch. The most suspects are "77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf" and https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438821452&sdata=ARiQNKn627v%2Bssux2ZF73YQDr1%2BEfXSYjteyN9LuMfw%3D&reserved=0>. Can you confirm you have those?

Do you have the log from "--log-level pmd.netvsc:debug" on the new failure?

Also, it seems you are running with "-vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1", I'm not entirely sure how netvsc pmd is involved.

Can you provide some details on your setup? I'll try to repro it to see if I can hit the failure.


________________________________
From: Dey, Souvik <sodey@rbbn.com>
Sent: Wednesday, September 16, 2020 3:07 PM
To: Long Li <longli@microsoft.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org <dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9


Looks like the below patch has changes a lot in the tx path which came in 18.11.9. Verified till 18.11.8 everything is working as expected.



https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438821452&sdata=ARiQNKn627v%2Bssux2ZF73YQDr1%2BEfXSYjteyN9LuMfw%3D&reserved=0>



Can you confirm if this change or the recv change patch proposed below will require any specific device config change as my same app does not work any longer.



--

Regards,

Souvik



From: Dey, Souvik
Sent: Wednesday, September 16, 2020 4:32 PM
To: Long Li <longli@microsoft.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



If it helps the tx_conf->tx_free_thresh is set to 0 from app side and nb_desc is set to 4096 from the app. The same was set in 18.11.6 too.



From: Dey, Souvik
Sent: Wednesday, September 16, 2020 3:44 PM
To: 'Long Li' <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Hi Long,



I did take this patch on top of 18.11.10rc1 , but with this patch and also with 18.11.10rc1 files, the rx itself is not working and dhcp fails. With 18.11.9 the rx is working fine and changes present in 18.11.10rc1(hn_rxtx.c  ) does not look to work. 18.11.6 was all proper but 18.11.9 is not helping. I am not using testpmd but using our own dpdk app on Azure.

We are using the below command line args:

-c 0xe -n 3 -w 83f0:00:02.0 -w 8cb0:00:02.0 --vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1 --log-level pmd.netvsc:debug --proc-type=primary



Thanks in advance for all the help.



--

Regards,

Souvik



From: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>
Sent: Wednesday, September 16, 2020 1:44 PM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________



Hi Souvik,



Please also pick up this patch:

https://patchwork.dpdk.org/patch/75341/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.dpdk.org%2Fpatch%2F75341&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438831444&sdata=28mY1tDVeEKPAze43jtHtz8EUKZWKBItFQfNPrY50B4%3D&reserved=0>



This patch didn’t make it to 18.11.



Let me know if you are still seeing failures. If you can reproduce the failure with testpmd or other tools in DPDK, please also share the command you used.



Thanks,

Long



From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 7:41 AM
To: Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Yes the patch solves the seg fault issue, but even after backporting the patch we are not able to send packets with size higher than 512. The packets not being transmitted is the real problem here. I have also tried to take the 18.11.10rc1 build but faced similar issues in transmitting the packets.



--

Regards,
Souvik



From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Kevin Traynor
Sent: Wednesday, September 16, 2020 10:18 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; longli@microsoft.com<mailto:longli@microsoft.com>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
> I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438831444&sdata=5%2BVAfNJ8hmylhhCV1lwnIwa%2BX0uA3uo7Ewt5SnimDrE%3D&reserved=0> . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox.dpdk.org%2Fstable%2F20200901100115.72365-1-ktraynor%40redhat.com&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438841439&sdata=mmMI5ZuEHEKGC4hs6X7oPbVPzs8IAyflffTYCuEq8%2Bc%3D&reserved=0>

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides-18.11%2Frel_notes%2Frelease_18_11.html&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438841439&sdata=crwb8l7Pg2hjNySymrqSTvBZcQdC38EfAjcQqWiK3M0%3D&reserved=0>

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
>
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75118&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438851435&sdata=DUAQbCT4BhZiWftmBKwywVLDzg0fjSK3ZZordpaGub4%3D&reserved=0>> fix in but nothing changed.
>
> --
> Regards,
> Souvik
>
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
>



________________________________

Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.

________________________________

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-17  6:57             ` Long Li
@ 2020-09-17 14:54               ` Dey, Souvik
  2020-09-18  7:08                 ` Long Li
  0 siblings, 1 reply; 12+ messages in thread
From: Dey, Souvik @ 2020-09-17 14:54 UTC (permalink / raw)
  To: Long Li, Kevin Traynor, dev
  Cc: Stephen Hemminger, Yigit, Ferruh, Stephen Hemminger

I have taken all the available approved patches from DPDK patchwork, but still Rx is not working properly.

A bit of summary:
With 18.11.6 – everything works fine.
With 18.11.9 –

  1.  Seg fault in tx path. Ported the https://patches.dpdk.org/patch/75001/ patch to fix that.
  2.  Large tx packets fail to go out with the following error - hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2. Looks like the changes done in the https://patches.dpdk.org/patch/67511/ is causing the issue.
Tried will all the available approved patches in dpdk on top on 18.11.9 –

  1.  RX stopped working . Packets comes up to the dpdk app but is not delivered to the kernel through kni interface. So could not test tx patch as dhcp is failing. I believe the patch https://patches.dpdk.org/patch/75341/ might be causing this.

It looks like definitely the patches are proper but something is wrong in the setup or the way packets are following. Just that everything worked fine till 18.11.8.
My setup is that

Kernel === kni interface === dpdk app === pmd .

Any further insight will be really valuable.. Thanks.

--
Regards,
Souvik

From: Long Li <longli@microsoft.com>
Sent: Thursday, September 17, 2020 2:58 AM
To: Dey, Souvik <sodey@rbbn.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

The patch below is correct. I'm wondering if you have missed another patch. The most suspects are "77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf" and https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438821452&sdata=ARiQNKn627v%2Bssux2ZF73YQDr1%2BEfXSYjteyN9LuMfw%3D&reserved=0>. Can you confirm you have those?

Do you have the log from "--log-level pmd.netvsc:debug" on the new failure?

Also, it seems you are running with "-vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1", I'm not entirely sure how netvsc pmd is involved.

Can you provide some details on your setup? I'll try to repro it to see if I can hit the failure.


________________________________
From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 3:07 PM
To: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9


Looks like the below patch has changes a lot in the tx path which came in 18.11.9. Verified till 18.11.8 everything is working as expected.



https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438821452&sdata=ARiQNKn627v%2Bssux2ZF73YQDr1%2BEfXSYjteyN9LuMfw%3D&reserved=0>



Can you confirm if this change or the recv change patch proposed below will require any specific device config change as my same app does not work any longer.



--

Regards,

Souvik



From: Dey, Souvik
Sent: Wednesday, September 16, 2020 4:32 PM
To: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



If it helps the tx_conf->tx_free_thresh is set to 0 from app side and nb_desc is set to 4096 from the app. The same was set in 18.11.6 too.



From: Dey, Souvik
Sent: Wednesday, September 16, 2020 3:44 PM
To: 'Long Li' <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Hi Long,



I did take this patch on top of 18.11.10rc1 , but with this patch and also with 18.11.10rc1 files, the rx itself is not working and dhcp fails. With 18.11.9 the rx is working fine and changes present in 18.11.10rc1(hn_rxtx.c  ) does not look to work. 18.11.6 was all proper but 18.11.9 is not helping. I am not using testpmd but using our own dpdk app on Azure.

We are using the below command line args:

-c 0xe -n 3 -w 83f0:00:02.0 -w 8cb0:00:02.0 --vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1 --log-level pmd.netvsc:debug --proc-type=primary



Thanks in advance for all the help.



--

Regards,

Souvik



From: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>
Sent: Wednesday, September 16, 2020 1:44 PM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________



Hi Souvik,



Please also pick up this patch:

https://patchwork.dpdk.org/patch/75341/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.dpdk.org%2Fpatch%2F75341&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438831444&sdata=28mY1tDVeEKPAze43jtHtz8EUKZWKBItFQfNPrY50B4%3D&reserved=0>



This patch didn’t make it to 18.11.



Let me know if you are still seeing failures. If you can reproduce the failure with testpmd or other tools in DPDK, please also share the command you used.



Thanks,

Long



From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 7:41 AM
To: Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Yes the patch solves the seg fault issue, but even after backporting the patch we are not able to send packets with size higher than 512. The packets not being transmitted is the real problem here. I have also tried to take the 18.11.10rc1 build but faced similar issues in transmitting the packets.



--

Regards,
Souvik



From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Kevin Traynor
Sent: Wednesday, September 16, 2020 10:18 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; longli@microsoft.com<mailto:longli@microsoft.com>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
> I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438831444&sdata=5%2BVAfNJ8hmylhhCV1lwnIwa%2BX0uA3uo7Ewt5SnimDrE%3D&reserved=0> . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox.dpdk.org%2Fstable%2F20200901100115.72365-1-ktraynor%40redhat.com&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438841439&sdata=mmMI5ZuEHEKGC4hs6X7oPbVPzs8IAyflffTYCuEq8%2Bc%3D&reserved=0>

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides-18.11%2Frel_notes%2Frelease_18_11.html&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438841439&sdata=crwb8l7Pg2hjNySymrqSTvBZcQdC38EfAjcQqWiK3M0%3D&reserved=0>

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
>
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75118&data=02%7C01%7Clongli%40microsoft.com%7C0b9366f76a924fbfa06c08d85a8d1e0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637358909438851435&sdata=DUAQbCT4BhZiWftmBKwywVLDzg0fjSK3ZZordpaGub4%3D&reserved=0>> fix in but nothing changed.
>
> --
> Regards,
> Souvik
>
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
>



________________________________

Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.

________________________________

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-17 14:54               ` Dey, Souvik
@ 2020-09-18  7:08                 ` Long Li
  2020-09-18 12:20                   ` Dey, Souvik
  0 siblings, 1 reply; 12+ messages in thread
From: Long Li @ 2020-09-18  7:08 UTC (permalink / raw)
  To: Dey, Souvik, Kevin Traynor, dev
  Cc: Stephen Hemminger, Yigit, Ferruh, Stephen Hemminger

Can you apply the following patch to see if it helps?

diff --git a/drivers/net/netvsc/hn_rxtx.c b/drivers/net/netvsc/hn_rxtx.c
index 813f8c3cc..8359cc121 100644
--- a/drivers/net/netvsc/hn_rxtx.c
+++ b/drivers/net/netvsc/hn_rxtx.c
@@ -160,8 +160,8 @@ static void hn_txd_init(struct rte_mempool *mp __rte_unused,

        txd->queue_id = txq->queue_id;
        txd->chim_index = NVS_CHIM_IDX_INVALID;
-       txd->rndis_pkt = (struct rndis_packet_msg *)(char *)txq->tx_rndis
-               + idx * HN_RNDIS_PKT_ALIGNED;
+       txd->rndis_pkt = (struct rndis_packet_msg *)((char *)txq->tx_rndis
+               + idx * HN_RNDIS_PKT_ALIGNED);
 }

 int


________________________________
From: Dey, Souvik <sodey@rbbn.com>
Sent: Thursday, September 17, 2020 7:54 AM
To: Long Li <longli@microsoft.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org <dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9


I have taken all the available approved patches from DPDK patchwork, but still Rx is not working properly.



A bit of summary:

With 18.11.6 – everything works fine.

With 18.11.9 –

  1.  Seg fault in tx path. Ported the https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090710567&sdata=gBu6wsS%2F7X695EL9nkpWvjA0KunV0cHOvgJSrHjyGog%3D&reserved=0> patch to fix that.
  2.  Large tx packets fail to go out with the following error - hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2. Looks like the changes done in the https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090715556&sdata=P3WhHszzSZtY3x1uFCBif1urLsEV47hEQhPD5GFl2X0%3D&reserved=0> is causing the issue.

Tried will all the available approved patches in dpdk on top on 18.11.9 –

  1.  RX stopped working . Packets comes up to the dpdk app but is not delivered to the kernel through kni interface. So could not test tx patch as dhcp is failing. I believe the patch https://patches.dpdk.org/patch/75341/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75341%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090720550&sdata=AMtDlizGrElXSuGCoAv40cSTSyYsXlkoGUVTMZe69ko%3D&reserved=0> might be causing this.



It looks like definitely the patches are proper but something is wrong in the setup or the way packets are following. Just that everything worked fine till 18.11.8.

My setup is that



Kernel === kni interface === dpdk app === pmd .



Any further insight will be really valuable.. Thanks.



--

Regards,

Souvik



From: Long Li <longli@microsoft.com>
Sent: Thursday, September 17, 2020 2:58 AM
To: Dey, Souvik <sodey@rbbn.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________



The patch below is correct. I'm wondering if you have missed another patch. The most suspects are "77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf" and https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090730530&sdata=nGAMMx2eM8x%2FHGpQQh%2Ffyuyg4ap1pJyXFK2MBUykMrY%3D&reserved=0>. Can you confirm you have those?



Do you have the log from "--log-level pmd.netvsc:debug" on the new failure?



Also, it seems you are running with "-vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1", I'm not entirely sure how netvsc pmd is involved.



Can you provide some details on your setup? I'll try to repro it to see if I can hit the failure.





________________________________

From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 3:07 PM
To: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Looks like the below patch has changes a lot in the tx path which came in 18.11.9. Verified till 18.11.8 everything is working as expected.



https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090735520&sdata=wXCfUolG7ktOuOwuZipOvPLBuCZ4nEW6ZbDNzO3SPbA%3D&reserved=0>



Can you confirm if this change or the recv change patch proposed below will require any specific device config change as my same app does not work any longer.



--

Regards,

Souvik



From: Dey, Souvik
Sent: Wednesday, September 16, 2020 4:32 PM
To: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



If it helps the tx_conf->tx_free_thresh is set to 0 from app side and nb_desc is set to 4096 from the app. The same was set in 18.11.6 too.



From: Dey, Souvik
Sent: Wednesday, September 16, 2020 3:44 PM
To: 'Long Li' <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Hi Long,



I did take this patch on top of 18.11.10rc1 , but with this patch and also with 18.11.10rc1 files, the rx itself is not working and dhcp fails. With 18.11.9 the rx is working fine and changes present in 18.11.10rc1(hn_rxtx.c  ) does not look to work. 18.11.6 was all proper but 18.11.9 is not helping. I am not using testpmd but using our own dpdk app on Azure.

We are using the below command line args:

-c 0xe -n 3 -w 83f0:00:02.0 -w 8cb0:00:02.0 --vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1 --log-level pmd.netvsc:debug --proc-type=primary



Thanks in advance for all the help.



--

Regards,

Souvik



From: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>
Sent: Wednesday, September 16, 2020 1:44 PM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________



Hi Souvik,



Please also pick up this patch:

https://patchwork.dpdk.org/patch/75341/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.dpdk.org%2Fpatch%2F75341&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090740511&sdata=PX8jIPpQc5LTn25kkKHCRZpJc9ZYJ7c6OPYppFqWtdM%3D&reserved=0>



This patch didn’t make it to 18.11.



Let me know if you are still seeing failures. If you can reproduce the failure with testpmd or other tools in DPDK, please also share the command you used.



Thanks,

Long



From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 7:41 AM
To: Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Yes the patch solves the seg fault issue, but even after backporting the patch we are not able to send packets with size higher than 512. The packets not being transmitted is the real problem here. I have also tried to take the 18.11.10rc1 build but faced similar issues in transmitting the packets.



--

Regards,
Souvik



From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Kevin Traynor
Sent: Wednesday, September 16, 2020 10:18 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; longli@microsoft.com<mailto:longli@microsoft.com>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
> I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090750494&sdata=VSyFhK2bV7NCmPK3WLIPGuhfEt7yMmovE7n%2FaYHWqt0%3D&reserved=0> . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox.dpdk.org%2Fstable%2F20200901100115.72365-1-ktraynor%40redhat.com&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090755485&sdata=j3Xw02nzP1Gx%2FIWzfSigf3psW%2FoP586SpjXgogJGBvM%3D&reserved=0>

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides-18.11%2Frel_notes%2Frelease_18_11.html&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090760473&sdata=tzxceUhieEPvLvlhuHGoCItDlZOCsyObXXr8JO1K%2Fkc%3D&reserved=0>

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
>
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75118&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090770452&sdata=wPK2yvPnj3mnPEMwyOH5C%2F3L6OENfwA1N2IEpDDPuzY%3D&reserved=0>> fix in but nothing changed.
>
> --
> Regards,
> Souvik
>
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
>



________________________________

Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.

________________________________

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-18  7:08                 ` Long Li
@ 2020-09-18 12:20                   ` Dey, Souvik
  2020-09-18 19:05                     ` Long Li
  0 siblings, 1 reply; 12+ messages in thread
From: Dey, Souvik @ 2020-09-18 12:20 UTC (permalink / raw)
  To: Long Li, Kevin Traynor, dev
  Cc: Stephen Hemminger, Yigit, Ferruh, Stephen Hemminger

Hi Long,
      Yes the below patch helped in getting the tx path issue resolved in 18.11.9. Thank you very much for the help. I think it will be really good to have this patch include in 18.11.10 too.
Current status:
18.11.9 –
1. Required 2 patches on top of 18.11.9 to make it work.
      a. https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090710567&sdata=gBu6wsS%2F7X695EL9nkpWvjA0KunV0cHOvgJSrHjyGog%3D&reserved=0>
      b. the below patch.
18.11.10 –
      1. Still facing rx path issue as explained below, where the rx mbuf is not  making it through the kni interface. As RX path packets are not coming up to the kernel, dhcp is failing and could not test the tx path with big packets. Also Arp packets are failing to reach through the kni interface.

Again, thanks a ton for the 18.11.9 Tx path issue resolution.

--
Regards,
Souvik

From: Long Li <longli@microsoft.com>
Sent: Friday, September 18, 2020 3:09 AM
To: Dey, Souvik <sodey@rbbn.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

Can you apply the following patch to see if it helps?

diff --git a/drivers/net/netvsc/hn_rxtx.c b/drivers/net/netvsc/hn_rxtx.c
index 813f8c3cc..8359cc121 100644
--- a/drivers/net/netvsc/hn_rxtx.c
+++ b/drivers/net/netvsc/hn_rxtx.c
@@ -160,8 +160,8 @@ static void hn_txd_init(struct rte_mempool *mp __rte_unused,

        txd->queue_id = txq->queue_id;
        txd->chim_index = NVS_CHIM_IDX_INVALID;
-       txd->rndis_pkt = (struct rndis_packet_msg *)(char *)txq->tx_rndis
-               + idx * HN_RNDIS_PKT_ALIGNED;
+       txd->rndis_pkt = (struct rndis_packet_msg *)((char *)txq->tx_rndis
+               + idx * HN_RNDIS_PKT_ALIGNED);
 }

 int

________________________________
From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Thursday, September 17, 2020 7:54 AM
To: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9


I have taken all the available approved patches from DPDK patchwork, but still Rx is not working properly.



A bit of summary:

With 18.11.6 – everything works fine.

With 18.11.9 –

  1.  Seg fault in tx path. Ported the https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090710567&sdata=gBu6wsS%2F7X695EL9nkpWvjA0KunV0cHOvgJSrHjyGog%3D&reserved=0> patch to fix that.
  2.  Large tx packets fail to go out with the following error - hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2. Looks like the changes done in the https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090715556&sdata=P3WhHszzSZtY3x1uFCBif1urLsEV47hEQhPD5GFl2X0%3D&reserved=0> is causing the issue.

Tried will all the available approved patches in dpdk on top on 18.11.9 –

  1.  RX stopped working . Packets comes up to the dpdk app but is not delivered to the kernel through kni interface. So could not test tx patch as dhcp is failing. I believe the patch https://patches.dpdk.org/patch/75341/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75341%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090720550&sdata=AMtDlizGrElXSuGCoAv40cSTSyYsXlkoGUVTMZe69ko%3D&reserved=0> might be causing this.



It looks like definitely the patches are proper but something is wrong in the setup or the way packets are following. Just that everything worked fine till 18.11.8.

My setup is that



Kernel === kni interface === dpdk app === pmd .



Any further insight will be really valuable.. Thanks.



--

Regards,

Souvik



From: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>
Sent: Thursday, September 17, 2020 2:58 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________



The patch below is correct. I'm wondering if you have missed another patch. The most suspects are "77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf" and https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090730530&sdata=nGAMMx2eM8x%2FHGpQQh%2Ffyuyg4ap1pJyXFK2MBUykMrY%3D&reserved=0>. Can you confirm you have those?



Do you have the log from "--log-level pmd.netvsc:debug" on the new failure?



Also, it seems you are running with "-vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1", I'm not entirely sure how netvsc pmd is involved.



Can you provide some details on your setup? I'll try to repro it to see if I can hit the failure.





________________________________

From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 3:07 PM
To: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Looks like the below patch has changes a lot in the tx path which came in 18.11.9. Verified till 18.11.8 everything is working as expected.



https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090735520&sdata=wXCfUolG7ktOuOwuZipOvPLBuCZ4nEW6ZbDNzO3SPbA%3D&reserved=0>



Can you confirm if this change or the recv change patch proposed below will require any specific device config change as my same app does not work any longer.



--

Regards,

Souvik



From: Dey, Souvik
Sent: Wednesday, September 16, 2020 4:32 PM
To: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



If it helps the tx_conf->tx_free_thresh is set to 0 from app side and nb_desc is set to 4096 from the app. The same was set in 18.11.6 too.



From: Dey, Souvik
Sent: Wednesday, September 16, 2020 3:44 PM
To: 'Long Li' <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Hi Long,



I did take this patch on top of 18.11.10rc1 , but with this patch and also with 18.11.10rc1 files, the rx itself is not working and dhcp fails. With 18.11.9 the rx is working fine and changes present in 18.11.10rc1(hn_rxtx.c  ) does not look to work. 18.11.6 was all proper but 18.11.9 is not helping. I am not using testpmd but using our own dpdk app on Azure.

We are using the below command line args:

-c 0xe -n 3 -w 83f0:00:02.0 -w 8cb0:00:02.0 --vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1 --log-level pmd.netvsc:debug --proc-type=primary



Thanks in advance for all the help.



--

Regards,

Souvik



From: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>
Sent: Wednesday, September 16, 2020 1:44 PM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________



Hi Souvik,



Please also pick up this patch:

https://patchwork.dpdk.org/patch/75341/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.dpdk.org%2Fpatch%2F75341&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090740511&sdata=PX8jIPpQc5LTn25kkKHCRZpJc9ZYJ7c6OPYppFqWtdM%3D&reserved=0>



This patch didn’t make it to 18.11.



Let me know if you are still seeing failures. If you can reproduce the failure with testpmd or other tools in DPDK, please also share the command you used.



Thanks,

Long



From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 7:41 AM
To: Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Yes the patch solves the seg fault issue, but even after backporting the patch we are not able to send packets with size higher than 512. The packets not being transmitted is the real problem here. I have also tried to take the 18.11.10rc1 build but faced similar issues in transmitting the packets.



--

Regards,
Souvik



From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Kevin Traynor
Sent: Wednesday, September 16, 2020 10:18 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; longli@microsoft.com<mailto:longli@microsoft.com>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
> I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090750494&sdata=VSyFhK2bV7NCmPK3WLIPGuhfEt7yMmovE7n%2FaYHWqt0%3D&reserved=0> . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox.dpdk.org%2Fstable%2F20200901100115.72365-1-ktraynor%40redhat.com&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090755485&sdata=j3Xw02nzP1Gx%2FIWzfSigf3psW%2FoP586SpjXgogJGBvM%3D&reserved=0>

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides-18.11%2Frel_notes%2Frelease_18_11.html&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090760473&sdata=tzxceUhieEPvLvlhuHGoCItDlZOCsyObXXr8JO1K%2Fkc%3D&reserved=0>

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
>
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75118&data=02%7C01%7Clongli%40microsoft.com%7C7966b9e87a6948f2487a08d85b19e667%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637359514090770452&sdata=wPK2yvPnj3mnPEMwyOH5C%2F3L6OENfwA1N2IEpDDPuzY%3D&reserved=0>> fix in but nothing changed.
>
> --
> Regards,
> Souvik
>
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
>



________________________________

Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.

________________________________

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9
  2020-09-18 12:20                   ` Dey, Souvik
@ 2020-09-18 19:05                     ` Long Li
  0 siblings, 0 replies; 12+ messages in thread
From: Long Li @ 2020-09-18 19:05 UTC (permalink / raw)
  To: Dey, Souvik, Kevin Traynor, dev
  Cc: Stephen Hemminger, Yigit, Ferruh, Stephen Hemminger

Thanks for confirming the fix!

I'm investigating the issue you are seeing with KNI in 18.11.10. Is there an easy way to reproduce it? It would be great if you can provide steps on how you setup your system.

Long
________________________________
From: Dey, Souvik <sodey@rbbn.com>
Sent: Friday, September 18, 2020 5:20 AM
To: Long Li <longli@microsoft.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org <dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9


Hi Long,

      Yes the below patch helped in getting the tx path issue resolved in 18.11.9. Thank you very much for the help. I think it will be really good to have this patch include in 18.11.10 too.

Current status:

18.11.9 –

1. Required 2 patches on top of 18.11.9 to make it work.

      a. https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001%2F&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552808178&sdata=baqQhKJTbOr%2F8VtMETHPW1pGn2I7DxKJ9Nywvq9isNk%3D&reserved=0>

      b. the below patch.

18.11.10 –

      1. Still facing rx path issue as explained below, where the rx mbuf is not  making it through the kni interface. As RX path packets are not coming up to the kernel, dhcp is failing and could not test the tx path with big packets. Also Arp packets are failing to reach through the kni interface.



Again, thanks a ton for the 18.11.9 Tx path issue resolution.



--

Regards,

Souvik



From: Long Li <longli@microsoft.com>
Sent: Friday, September 18, 2020 3:09 AM
To: Dey, Souvik <sodey@rbbn.com>; Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org
Cc: Stephen Hemminger <stephen@networkplumber.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Stephen Hemminger <sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________



Can you apply the following patch to see if it helps?



diff --git a/drivers/net/netvsc/hn_rxtx.c b/drivers/net/netvsc/hn_rxtx.c

index 813f8c3cc..8359cc121 100644

--- a/drivers/net/netvsc/hn_rxtx.c

+++ b/drivers/net/netvsc/hn_rxtx.c

@@ -160,8 +160,8 @@ static void hn_txd_init(struct rte_mempool *mp __rte_unused,



        txd->queue_id = txq->queue_id;

        txd->chim_index = NVS_CHIM_IDX_INVALID;

-       txd->rndis_pkt = (struct rndis_packet_msg *)(char *)txq->tx_rndis

-               + idx * HN_RNDIS_PKT_ALIGNED;

+       txd->rndis_pkt = (struct rndis_packet_msg *)((char *)txq->tx_rndis

+               + idx * HN_RNDIS_PKT_ALIGNED);

 }



 int



________________________________

From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Thursday, September 17, 2020 7:54 AM
To: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



I have taken all the available approved patches from DPDK patchwork, but still Rx is not working properly.



A bit of summary:

With 18.11.6 – everything works fine.

With 18.11.9 –

  1.  Seg fault in tx path. Ported the https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001%2F&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552818173&sdata=2vMaj29sinRi%2B%2BY71%2FUImDcQXAajyFopB%2FIhfZK8B34%3D&reserved=0> patch to fix that.
  2.  Large tx packets fail to go out with the following error - hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2. Looks like the changes done in the https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552818173&sdata=zo3rkiUH5vDpUydYi1SEYvMNnT5UcOcHfSnyOc9KbIs%3D&reserved=0> is causing the issue.

Tried will all the available approved patches in dpdk on top on 18.11.9 –

  1.  RX stopped working . Packets comes up to the dpdk app but is not delivered to the kernel through kni interface. So could not test tx patch as dhcp is failing. I believe the patch https://patches.dpdk.org/patch/75341/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75341%2F&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552828170&sdata=4o8Hw%2BGpYXT4ZTS2mG9Vb02SbbjqHnDRheL4bjL1a04%3D&reserved=0> might be causing this.



It looks like definitely the patches are proper but something is wrong in the setup or the way packets are following. Just that everything worked fine till 18.11.8.

My setup is that



Kernel === kni interface === dpdk app === pmd .



Any further insight will be really valuable.. Thanks.



--

Regards,

Souvik



From: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>
Sent: Thursday, September 17, 2020 2:58 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________



The patch below is correct. I'm wondering if you have missed another patch. The most suspects are "77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf" and https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552828170&sdata=%2FJ82g1k5Nb6qe2jdja%2FZLiwAT9Q4sqziV%2FP5%2Fkt61zE%3D&reserved=0>. Can you confirm you have those?



Do you have the log from "--log-level pmd.netvsc:debug" on the new failure?



Also, it seems you are running with "-vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1", I'm not entirely sure how netvsc pmd is involved.



Can you provide some details on your setup? I'll try to repro it to see if I can hit the failure.





________________________________

From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 3:07 PM
To: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org> <dev@dpdk.org<mailto:dev@dpdk.org>>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Looks like the below patch has changes a lot in the tx path which came in 18.11.9. Verified till 18.11.8 everything is working as expected.



https://patches.dpdk.org/patch/67511/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F67511%2F&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552838161&sdata=wbQ5gbZREEOkz8IZETMQS8RCieLY06R5oFbIV1iCrvs%3D&reserved=0>



Can you confirm if this change or the recv change patch proposed below will require any specific device config change as my same app does not work any longer.



--

Regards,

Souvik



From: Dey, Souvik
Sent: Wednesday, September 16, 2020 4:32 PM
To: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



If it helps the tx_conf->tx_free_thresh is set to 0 from app side and nb_desc is set to 4096 from the app. The same was set in 18.11.6 too.



From: Dey, Souvik
Sent: Wednesday, September 16, 2020 3:44 PM
To: 'Long Li' <longli@microsoft.com<mailto:longli@microsoft.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Hi Long,



I did take this patch on top of 18.11.10rc1 , but with this patch and also with 18.11.10rc1 files, the rx itself is not working and dhcp fails. With 18.11.9 the rx is working fine and changes present in 18.11.10rc1(hn_rxtx.c  ) does not look to work. 18.11.6 was all proper but 18.11.9 is not helping. I am not using testpmd but using our own dpdk app on Azure.

We are using the below command line args:

-c 0xe -n 3 -w 83f0:00:02.0 -w 8cb0:00:02.0 --vdev=net_vdev_netvsc0,iface=mlxhvpkt0 --vdev=net_vdev_netvsc1,iface=mlxhvpkt1 --log-level pmd.netvsc:debug --proc-type=primary



Thanks in advance for all the help.



--

Regards,

Souvik



From: Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>
Sent: Wednesday, September 16, 2020 1:44 PM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________



Hi Souvik,



Please also pick up this patch:

https://patchwork.dpdk.org/patch/75341/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.dpdk.org%2Fpatch%2F75341&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552838161&sdata=arUEgY26O19gn8kVMUsSSWxzH0hs48cbcxfB5lu5WNs%3D&reserved=0>



This patch didn’t make it to 18.11.



Let me know if you are still seeing failures. If you can reproduce the failure with testpmd or other tools in DPDK, please also share the command you used.



Thanks,

Long



From: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>
Sent: Wednesday, September 16, 2020 7:41 AM
To: Kevin Traynor <ktraynor@redhat.com<mailto:ktraynor@redhat.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; Long Li <longli@microsoft.com<mailto:longli@microsoft.com>>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Stephen Hemminger <sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>>
Subject: RE: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



Yes the patch solves the seg fault issue, but even after backporting the patch we are not able to send packets with size higher than 512. The packets not being transmitted is the real problem here. I have also tried to take the 18.11.10rc1 build but faced similar issues in transmitting the packets.



--

Regards,
Souvik



From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Kevin Traynor
Sent: Wednesday, September 16, 2020 10:18 AM
To: Dey, Souvik <sodey@rbbn.com<mailto:sodey@rbbn.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>>; longli@microsoft.com<mailto:longli@microsoft.com>; Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; sthemmin@microsoft.com<mailto:sthemmin@microsoft.com>
Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9



________________________________

NOTICE: This email was received from an EXTERNAL sender

________________________________

On 16/09/2020 13:00, Dey, Souvik wrote:
> Hi All,
> I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75001&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552848155&sdata=sFqw2d9jAP1AsMTNMiIdpfE9ojCEGNBw1lklgn4oBto%3D&reserved=0> . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we

Hi,

The patch mentioned is merged to the 18.11 branch in dpdk-stable repo
and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September.

In case it's already fixed, you can try the 18.11.10-rc1 release. These
netvsc backports were added since 18.11.9. The corresponding dpdk main
commit id's are in the full log.

$ git log --oneline v18.11.9..HEAD .
363bdf0f49 net/netvsc: fix chimney index
5b3a327709 net/netvsc: fix crash during Tx
77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf
59257aaaa1 net/netvsc: do not spin forever waiting for reply

The list of fixes that were tagged for stable but not backported is
here:
http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox.dpdk.org%2Fstable%2F20200901100115.72365-1-ktraynor%40redhat.com&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552848155&sdata=B1XmVjrpxNPMwKEe7r3KWSnGpM80m0HiDnvAs9nLtNM%3D&reserved=0>

are all good but any larger packets/fragmented packets are getting
dropped after some time. As soon as we start to receive the transmit
completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater
than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less
than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why
this is happening and is there is some fix which is already there which
is done after 18.11.9 release ?

Maybe you can bisect the 18.11 branch to find where it stops working,
there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The
release notes [1] or git history will tell you which commits are in
which releases. Bear in mind that some commits were rebased for 18.11,
so if you find the offending commit, you might want to check that it was
correctly backported too.

Kevin.

[1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides-18.11%2Frel_notes%2Frelease_18_11.html&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552858150&sdata=4%2FBY5fpiw%2FOarx1WldI6LodMqyez%2B%2FctQ5lX5nc2BtA%3D&reserved=0>

> We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help.
>
> PS: I also tried to put the net/netvsc: return the correct chimney index <https://patches.dpdk.org/patch/75118/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F75118&data=02%7C01%7Clongli%40microsoft.com%7Cecb70cea719e4b29e24f08d85bcd4892%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637360284552858150&sdata=ZihqtPEA7gQpp38YqcgQqJgklp%2B%2FhMksYBTb2yWWGUI%3D&reserved=0>> fix in but nothing changed.
>
> --
> Regards,
> Souvik
>
> Log Snippet:
> I can see the below errors coming after some transmitting.
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_flush_txagg() tx: port 2:0 tx 2048 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2624 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514
> hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242
>
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_flush_txagg() tx: port 2:0 tx 2688 size 302
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242
> hn_flush_txagg() tx: port 2:0 tx 3264 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2
> hn_flush_txagg() tx: port 2:0 tx 3328 size 571
> hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2
> hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0
> hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2
> hn_flush_txagg() tx: port 2:0 tx 3392 size 120
> hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60
> hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0
>



________________________________

Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.

________________________________

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2020-09-18 19:05 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-16 12:00 [dpdk-dev] Issue with net/netvsc pmd in 18.11.9 Dey, Souvik
2020-09-16 14:18 ` Kevin Traynor
2020-09-16 14:41   ` Dey, Souvik
2020-09-16 17:43     ` Long Li
2020-09-16 19:43       ` Dey, Souvik
2020-09-16 20:32         ` Dey, Souvik
2020-09-16 22:07           ` Dey, Souvik
2020-09-17  6:57             ` Long Li
2020-09-17 14:54               ` Dey, Souvik
2020-09-18  7:08                 ` Long Li
2020-09-18 12:20                   ` Dey, Souvik
2020-09-18 19:05                     ` Long Li

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).