[dpdk-dev] queue to VF assigment in SR-IOV

[dpdk-dev] queue to VF assigment in SR-IOV

[Mauricio Vásquez]

Dear All,

I implemented a program that uses flow director to forward packets to a
specific virtual function, however I faced the problem that I did not know
which queue belongs to a VF. I found in [1] that in the case of Intel
82599, queues 0-7 belongs to VF0, 8-15 to VF1 and so on, I tested it but it
did not work, using the trial and error method I found that queue 0 is in
VF0, queue 4 in VF1 and so on.

My question is: is there a standard way to know which queues belong to a
specific VF?

Thanks in advance

Mauricio V,

[1]
http://www.intel.it/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf,
Table 7-72

Re: [dpdk-dev] queue to VF assigment in SR-IOV

[Alexander Duyck]

On Tue, Jun 7, 2016 at 1:49 PM, Mauricio Vásquez
<mauricio.vasquezbernal@studenti.polito.it> wrote:
> Dear All,
>
> I implemented a program that uses flow director to forward packets to a
> specific virtual function, however I faced the problem that I did not know
> which queue belongs to a VF. I found in [1] that in the case of Intel
> 82599, queues 0-7 belongs to VF0, 8-15 to VF1 and so on, I tested it but it
> did not work, using the trial and error method I found that queue 0 is in
> VF0, queue 4 in VF1 and so on.
>
> My question is: is there a standard way to know which queues belong to a
> specific VF?
>
> Thanks in advance
>
> Mauricio V,
>
> [1]
> http://www.intel.it/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf,
> Table 7-72

If you are using the kernel driver the way the queues are laid out
depends on the number of VFs allocated and what features are enabled
in the kernel.  Assuming you are not using DCB you should be able to
figure out how many queues are being allocated via VF by looking at
the output of "ethtool -l <iface>".  The upper limit on RSS is t he
number of queues each pool is allocated.

So for example if you only enable up to 31 VFs then the PF driver
allocates 4 queues per VF so you would have queues 0 - 3 allocated to
VF0, queues 4-7 allocated to VF1, etc all the way through to the PF
occupying (num_vfs * 4) to 127.  If you enable 32 or more VFs then the
number of queues drops to 2 per VF and RSS on the PF will be limited
to the 2 queues following the block reserved for the VFs.

There are a few other configurations such as if DCB is enabled I
believe it is possible to get 8 queues per VF if less than 16 VFs are
allocated but in such a case you would not have access to RSS.  In
this case if the maximum combined queue count reported is 1 you would
need to check to see how many TCs are being supported by the PF in
order to determine if the queue count is 4 or 8 per VF.

- Alex

Re: [dpdk-dev] queue to VF assigment in SR-IOV

[Mauricio Vásquez]

Hello Alexander,

On Tue, Jun 7, 2016 at 11:31 PM, Alexander Duyck <alexander.duyck@gmail.com>
wrote:

> On Tue, Jun 7, 2016 at 1:49 PM, Mauricio Vásquez
> <mauricio.vasquezbernal@studenti.polito.it> wrote:
> > Dear All,
> >
> > I implemented a program that uses flow director to forward packets to a
> > specific virtual function, however I faced the problem that I did not
> know
> > which queue belongs to a VF. I found in [1] that in the case of Intel
> > 82599, queues 0-7 belongs to VF0, 8-15 to VF1 and so on, I tested it but
> it
> > did not work, using the trial and error method I found that queue 0 is in
> > VF0, queue 4 in VF1 and so on.
> >
> > My question is: is there a standard way to know which queues belong to a
> > specific VF?
> >
> > Thanks in advance
> >
> > Mauricio V,
> >
> > [1]
> >
> http://www.intel.it/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf
> ,
> > Table 7-72
>
> If you are using the kernel driver the way the queues are laid out
> depends on the number of VFs allocated and what features are enabled
> in the kernel.


I forgot to mention that I am using the DPDK ixgbe PMD.


> Assuming you are not using DCB you should be able to
> figure out how many queues are being allocated via VF by looking at
> the output of "ethtool -l <iface>".  The upper limit on RSS is t he
> number of queues each pool is allocated.
>
> So for example if you only enable up to 31 VFs then the PF driver
> allocates 4 queues per VF so you would have queues 0 - 3 allocated to
> VF0, queues 4-7 allocated to VF1, etc all the way through to the PF
> occupying (num_vfs * 4) to 127.  If you enable 32 or more VFs then the
> number of queues drops to 2 per VF and RSS on the PF will be limited
> to the 2 queues following the block reserved for the VFs.
>
>
I found that the behavior of the DPDK PMD is almost the same you described:
1 - 15 VFs -> 8 queues per VF
16 - 31 VFs -> 4 queues per VF
>= 32 VFs -> 2 queues per VF

But, according to the datasheet it should be
16 VFs -> 8 queues per VF
32 VFs -> 4 queues per VF
64 VFs -> 2 queues per VF

Am I missing something?

One extra thing that I am not understanding, in the case I assign the
maximum number of possible VFs, the PF remains without queues?

There are a few other configurations such as if DCB is enabled I
> believe it is possible to get 8 queues per VF if less than 16 VFs are
> allocated but in such a case you would not have access to RSS.  In
> this case if the maximum combined queue count reported is 1 you would
> need to check to see how many TCs are being supported by the PF in
> order to determine if the queue count is 4 or 8 per VF.
>
> - Alex
>

Thank you very much,

Mauricio V,

Re: [dpdk-dev] queue to VF assigment in SR-IOV

[Alexander Duyck]

On Mon, Jun 13, 2016 at 4:56 AM, Mauricio Vásquez
<mauricio.vasquezbernal@studenti.polito.it> wrote:
> Hello Alexander,
>
> On Tue, Jun 7, 2016 at 11:31 PM, Alexander Duyck <alexander.duyck@gmail.com>
> wrote:
>>
>> On Tue, Jun 7, 2016 at 1:49 PM, Mauricio Vásquez
>> <mauricio.vasquezbernal@studenti.polito.it> wrote:
>> > Dear All,
>> >
>> > I implemented a program that uses flow director to forward packets to a
>> > specific virtual function, however I faced the problem that I did not
>> > know
>> > which queue belongs to a VF. I found in [1] that in the case of Intel
>> > 82599, queues 0-7 belongs to VF0, 8-15 to VF1 and so on, I tested it but
>> > it
>> > did not work, using the trial and error method I found that queue 0 is
>> > in
>> > VF0, queue 4 in VF1 and so on.
>> >
>> > My question is: is there a standard way to know which queues belong to a
>> > specific VF?
>> >
>> > Thanks in advance
>> >
>> > Mauricio V,
>> >
>> > [1]
>> >
>> > http://www.intel.it/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf,
>> > Table 7-72
>>
>> If you are using the kernel driver the way the queues are laid out
>> depends on the number of VFs allocated and what features are enabled
>> in the kernel.
>
>
> I forgot to mention that I am using the DPDK ixgbe PMD.
>
>>
>> Assuming you are not using DCB you should be able to
>> figure out how many queues are being allocated via VF by looking at
>> the output of "ethtool -l <iface>".  The upper limit on RSS is t he
>> number of queues each pool is allocated.
>>
>> So for example if you only enable up to 31 VFs then the PF driver
>> allocates 4 queues per VF so you would have queues 0 - 3 allocated to
>> VF0, queues 4-7 allocated to VF1, etc all the way through to the PF
>> occupying (num_vfs * 4) to 127.  If you enable 32 or more VFs then the
>> number of queues drops to 2 per VF and RSS on the PF will be limited
>> to the 2 queues following the block reserved for the VFs.
>>
>
> I found that the behavior of the DPDK PMD is almost the same you described:
> 1 - 15 VFs -> 8 queues per VF
> 16 - 31 VFs -> 4 queues per VF
>>= 32 VFs -> 2 queues per VF
>
> But, according to the datasheet it should be
> 16 VFs -> 8 queues per VF
> 32 VFs -> 4 queues per VF
> 64 VFs -> 2 queues per VF
>
> Am I missing something?

The datasheet should be referring to "VM pools".  The PF consumes one
pool for any queues it is using.  As such VFs + 1 is the total number
of pools in use if the PF is active.

> One extra thing that I am not understanding, in the case I assign the
> maximum number of possible VFs, the PF remains without queues?

The device can support at most 64 pools.  So if you are allocating 64
VFs then there are no resources left for the PF to allocate queues
from.

I hope this helps to make it a bit clearer.

- Alex

Re: [dpdk-dev] queue to VF assigment in SR-IOV

[Mauricio Vásquez]

Hello Alexander,


On Mon, Jun 13, 2016 at 6:02 PM, Alexander Duyck <alexander.duyck@gmail.com>
wrote:

> On Mon, Jun 13, 2016 at 4:56 AM, Mauricio Vásquez
> <mauricio.vasquezbernal@studenti.polito.it> wrote:
> > Hello Alexander,
> >
> > On Tue, Jun 7, 2016 at 11:31 PM, Alexander Duyck <
> alexander.duyck@gmail.com>
> > wrote:
> >>
> >> On Tue, Jun 7, 2016 at 1:49 PM, Mauricio Vásquez
> >> <mauricio.vasquezbernal@studenti.polito.it> wrote:
> >> > Dear All,
> >> >
> >> > I implemented a program that uses flow director to forward packets to
> a
> >> > specific virtual function, however I faced the problem that I did not
> >> > know
> >> > which queue belongs to a VF. I found in [1] that in the case of Intel
> >> > 82599, queues 0-7 belongs to VF0, 8-15 to VF1 and so on, I tested it
> but
> >> > it
> >> > did not work, using the trial and error method I found that queue 0 is
> >> > in
> >> > VF0, queue 4 in VF1 and so on.
> >> >
> >> > My question is: is there a standard way to know which queues belong
> to a
> >> > specific VF?
> >> >
> >> > Thanks in advance
> >> >
> >> > Mauricio V,
> >> >
> >> > [1]
> >> >
> >> >
> http://www.intel.it/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf
> ,
> >> > Table 7-72
> >>
> >> If you are using the kernel driver the way the queues are laid out
> >> depends on the number of VFs allocated and what features are enabled
> >> in the kernel.
> >
> >
> > I forgot to mention that I am using the DPDK ixgbe PMD.
> >
> >>
> >> Assuming you are not using DCB you should be able to
> >> figure out how many queues are being allocated via VF by looking at
> >> the output of "ethtool -l <iface>".  The upper limit on RSS is t he
> >> number of queues each pool is allocated.
> >>
> >> So for example if you only enable up to 31 VFs then the PF driver
> >> allocates 4 queues per VF so you would have queues 0 - 3 allocated to
> >> VF0, queues 4-7 allocated to VF1, etc all the way through to the PF
> >> occupying (num_vfs * 4) to 127.  If you enable 32 or more VFs then the
> >> number of queues drops to 2 per VF and RSS on the PF will be limited
> >> to the 2 queues following the block reserved for the VFs.
> >>
> >
> > I found that the behavior of the DPDK PMD is almost the same you
> described:
> > 1 - 15 VFs -> 8 queues per VF
> > 16 - 31 VFs -> 4 queues per VF
> >>= 32 VFs -> 2 queues per VF
> >
> > But, according to the datasheet it should be
> > 16 VFs -> 8 queues per VF
> > 32 VFs -> 4 queues per VF
> > 64 VFs -> 2 queues per VF
> >
> > Am I missing something?
>
> The datasheet should be referring to "VM pools".  The PF consumes one
> pool for any queues it is using.  As such VFs + 1 is the total number
> of pools in use if the PF is active.
>
> > One extra thing that I am not understanding, in the case I assign the
> > maximum number of possible VFs, the PF remains without queues?
>
> The device can support at most 64 pools.  So if you are allocating 64
> VFs then there are no resources left for the PF to allocate queues
> from.
>
> I hope this helps to make it a bit clearer.
>

This made everything clear for me, thanks.


>
> - Alex
>

Mauricio V,