[dpdk-users] Hyper-V multi-queue not working on Kernel v4.20

[dpdk-users] Hyper-V multi-queue not working on Kernel v4.20

[Gareth Bradshaw]

Hi,

I'm developing a DPDK application for Hyper-V and would like to use Kernel v4.20 as it fixes a crash in previous kernels (per this pull request<https://github.com/torvalds/linux/commit/cdfa835c6e5e87d145f9f632b58843de97509f2b>) but multi-queue support seems to have broken in that kernel.  It works in kernels 4.17, 4.18 and 4.19.

Using testpmd with parameters "--forward-mode=txonly --nb-cores=3 --txq=3 --rxq=3" I get the following error "vmbus_uio_get_subchan(): invalid subchannel id 0".

Tracing the code a bit I find that it's trying to read /sys/bus/devices/<devid>/channels/21/subchannel_id but when I try to cat the same file I get the error "Invalid argument".  I've compared to previous kernel versions and see the following differences:

  *   When the VM boots, the device is bound to the kernel driver and there are 4 channels on both kernel versions.  It's a 4 core VM.
  *   When I bind the device to uio_hv_generic, kernel v.19.5 (and previous) has a single channel and it has a valid subchannel_id.  On kernel 4.20 the 4 channel folders remain but they are all invalid.
  *   When I start testpmd with 3 queues, on kernel v4.19.5 two additional channels are created, they're valid and testpmd works as expected.  On kernel 4.20 the 4 invalid channels remain invalid and testpmd gives the error I mentioned.

Any help getting this working would be greatly appreciated.

Thanks
Gareth

Re: [dpdk-users] Hyper-V multi-queue not working on Kernel v4.20

[Stephen Hemminger]

On Wed, 5 Dec 2018 11:05:58 +0000
Gareth Bradshaw <gareth@vectra.ai> wrote:

> Hi,
> 
> I'm developing a DPDK application for Hyper-V and would like to use Kernel v4.20 as it fixes a crash in previous kernels (per this pull request<https://github.com/torvalds/linux/commit/cdfa835c6e5e87d145f9f632b58843de97509f2b>) but multi-queue support seems to have broken in that kernel.  It works in kernels 4.17, 4.18 and 4.19.
> 
> Using testpmd with parameters "--forward-mode=txonly --nb-cores=3 --txq=3 --rxq=3" I get the following error "vmbus_uio_get_subchan(): invalid subchannel id 0".
> 
> Tracing the code a bit I find that it's trying to read /sys/bus/devices/<devid>/channels/21/subchannel_id but when I try to cat the same file I get the error "Invalid argument".  I've compared to previous kernel versions and see the following differences:
> 
>   *   When the VM boots, the device is bound to the kernel driver and there are 4 channels on both kernel versions.  It's a 4 core VM.
>   *   When I bind the device to uio_hv_generic, kernel v.19.5 (and previous) has a single channel and it has a valid subchannel_id.  On kernel 4.20 the 4 channel folders remain but they are all invalid.
>   *   When I start testpmd with 3 queues, on kernel v4.19.5 two additional channels are created, they're valid and testpmd works as expected.  On kernel 4.20 the 4 invalid channels remain invalid and testpmd gives the error I mentioned.
> 
> Any help getting this working would be greatly appreciated.
> 
> Thanks
> Gareth
> 
> 
> 
> 
> 
> 

There is a race that shows up in sub channel initialization, it is being worked on the development
mailing list. No resolution yet.

Re: [dpdk-users] Hyper-V multi-queue not working on Kernel v4.20

[Gareth Bradshaw]

Thanks for coming back to me Stephen.  I found the thread you mentioned and will keep an eye on it.  I'm happy to help out with testing the change.

Gareth


-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org> 
Sent: Wednesday 5 December 2018 16:04
To: Gareth Bradshaw <gareth@vectra.ai>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Hyper-V multi-queue not working on Kernel v4.20

On Wed, 5 Dec 2018 11:05:58 +0000
Gareth Bradshaw <gareth@vectra.ai> wrote:

> Hi,
> 
> I'm developing a DPDK application for Hyper-V and would like to use Kernel v4.20 as it fixes a crash in previous kernels (per this pull request<https://github.com/torvalds/linux/commit/cdfa835c6e5e87d145f9f632b58843de97509f2b>) but multi-queue support seems to have broken in that kernel.  It works in kernels 4.17, 4.18 and 4.19.
> 
> Using testpmd with parameters "--forward-mode=txonly --nb-cores=3 --txq=3 --rxq=3" I get the following error "vmbus_uio_get_subchan(): invalid subchannel id 0".
> 
> Tracing the code a bit I find that it's trying to read /sys/bus/devices/<devid>/channels/21/subchannel_id but when I try to cat the same file I get the error "Invalid argument".  I've compared to previous kernel versions and see the following differences:
> 
>   *   When the VM boots, the device is bound to the kernel driver and there are 4 channels on both kernel versions.  It's a 4 core VM.
>   *   When I bind the device to uio_hv_generic, kernel v.19.5 (and previous) has a single channel and it has a valid subchannel_id.  On kernel 4.20 the 4 channel folders remain but they are all invalid.
>   *   When I start testpmd with 3 queues, on kernel v4.19.5 two additional channels are created, they're valid and testpmd works as expected.  On kernel 4.20 the 4 invalid channels remain invalid and testpmd gives the error I mentioned.
> 
> Any help getting this working would be greatly appreciated.
> 
> Thanks
> Gareth
> 
> 
> 
> 
> 
> 

There is a race that shows up in sub channel initialization, it is being worked on the development mailing list. No resolution yet.

Re: [dpdk-users] Hyper-V multi-queue not working on Kernel v4.20

[Gareth Bradshaw]

Hi Stephen,

I wanted to check in to see how the fix is going.  I've repro'd the patches yourself and Mohammed Gamal were discussing on the development mailing list and see the same results, i.e. multi-channel works the first time testpmd is launched but subsequent launches result in an error setting up the sub-channels.  I'm on the development mailing list so if you'd like to add me to the thread I'll be happy to test the patches.

Cheers,
Gareth


-----Original Message-----
From: Gareth Bradshaw 
Sent: Wednesday 5 December 2018 17:10
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: users@dpdk.org
Subject: RE: [dpdk-users] Hyper-V multi-queue not working on Kernel v4.20

Thanks for coming back to me Stephen.  I found the thread you mentioned and will keep an eye on it.  I'm happy to help out with testing the change.

Gareth


-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org> 
Sent: Wednesday 5 December 2018 16:04
To: Gareth Bradshaw <gareth@vectra.ai>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Hyper-V multi-queue not working on Kernel v4.20

On Wed, 5 Dec 2018 11:05:58 +0000
Gareth Bradshaw <gareth@vectra.ai> wrote:

> Hi,
> 
> I'm developing a DPDK application for Hyper-V and would like to use Kernel v4.20 as it fixes a crash in previous kernels (per this pull request<https://github.com/torvalds/linux/commit/cdfa835c6e5e87d145f9f632b58843de97509f2b>) but multi-queue support seems to have broken in that kernel.  It works in kernels 4.17, 4.18 and 4.19.
> 
> Using testpmd with parameters "--forward-mode=txonly --nb-cores=3 --txq=3 --rxq=3" I get the following error "vmbus_uio_get_subchan(): invalid subchannel id 0".
> 
> Tracing the code a bit I find that it's trying to read /sys/bus/devices/<devid>/channels/21/subchannel_id but when I try to cat the same file I get the error "Invalid argument".  I've compared to previous kernel versions and see the following differences:
> 
>   *   When the VM boots, the device is bound to the kernel driver and there are 4 channels on both kernel versions.  It's a 4 core VM.
>   *   When I bind the device to uio_hv_generic, kernel v.19.5 (and previous) has a single channel and it has a valid subchannel_id.  On kernel 4.20 the 4 channel folders remain but they are all invalid.
>   *   When I start testpmd with 3 queues, on kernel v4.19.5 two additional channels are created, they're valid and testpmd works as expected.  On kernel 4.20 the 4 invalid channels remain invalid and testpmd gives the error I mentioned.
> 
> Any help getting this working would be greatly appreciated.
> 
> Thanks
> Gareth
> 
> 
> 
> 
> 
> 

There is a race that shows up in sub channel initialization, it is being worked on the development mailing list. No resolution yet.

Re: [dpdk-users] Hyper-V multi-queue not working on Kernel v4.20

[Stephen Hemminger]

On Thu, 13 Dec 2018 16:48:55 +0000
Gareth Bradshaw <gareth@vectra.ai> wrote:

> Hi Stephen,
> 
> I wanted to check in to see how the fix is going.  I've repro'd the patches yourself and Mohammed Gamal were discussing on the development mailing list and see the same results, i.e. multi-channel works the first time testpmd is launched but subsequent launches result in an error setting up the sub-channels.  I'm on the development mailing list so if you'd like to add me to the thread I'll be happy to test the patches.
> 
> Cheers,
> Gareth
> 
> 
> -----Original Message-----
> From: Gareth Bradshaw 
> Sent: Wednesday 5 December 2018 17:10
> To: Stephen Hemminger <stephen@networkplumber.org>
> Cc: users@dpdk.org
> Subject: RE: [dpdk-users] Hyper-V multi-queue not working on Kernel v4.20
> 
> Thanks for coming back to me Stephen.  I found the thread you mentioned and will keep an eye on it.  I'm happy to help out with testing the change.
> 
> Gareth
> 
> 
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org> 
> Sent: Wednesday 5 December 2018 16:04
> To: Gareth Bradshaw <gareth@vectra.ai>
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] Hyper-V multi-queue not working on Kernel v4.20
> 
> On Wed, 5 Dec 2018 11:05:58 +0000
> Gareth Bradshaw <gareth@vectra.ai> wrote:
> 
> > Hi,
> > 
> > I'm developing a DPDK application for Hyper-V and would like to use Kernel v4.20 as it fixes a crash in previous kernels (per this pull request<https://github.com/torvalds/linux/commit/cdfa835c6e5e87d145f9f632b58843de97509f2b>) but multi-queue support seems to have broken in that kernel.  It works in kernels 4.17, 4.18 and 4.19.
> > 
> > Using testpmd with parameters "--forward-mode=txonly --nb-cores=3 --txq=3 --rxq=3" I get the following error "vmbus_uio_get_subchan(): invalid subchannel id 0".
> > 
> > Tracing the code a bit I find that it's trying to read /sys/bus/devices/<devid>/channels/21/subchannel_id but when I try to cat the same file I get the error "Invalid argument".  I've compared to previous kernel versions and see the following differences:
> > 
> >   *   When the VM boots, the device is bound to the kernel driver and there are 4 channels on both kernel versions.  It's a 4 core VM.
> >   *   When I bind the device to uio_hv_generic, kernel v.19.5 (and previous) has a single channel and it has a valid subchannel_id.  On kernel 4.20 the 4 channel folders remain but they are all invalid.
> >   *   When I start testpmd with 3 queues, on kernel v4.19.5 two additional channels are created, they're valid and testpmd works as expected.  On kernel 4.20 the 4 invalid channels remain invalid and testpmd gives the error I mentioned.
> > 
> > Any help getting this working would be greatly appreciated.
> > 
> > Thanks
> > Gareth
> > 
> > 
> > 
> > 
> > 
> >   
> 
> There is a race that shows up in sub channel initialization, it is being worked on the development mailing list. No resolution yet.

There were two kernel patches:
http://driverdev.linuxdriverproject.org/pipermail/driverdev-devel/2018-December/129607.html

http://driverdev.linuxdriverproject.org/pipermail/driverdev-devel/2018-December/129656.html

After that it works fine.