[dpdk-dev] [PATCH] bus/pci: support segment value as address domain on Windows

[dpdk-dev] [PATCH] bus/pci: support segment value as address domain on Windows

[Tal Shnaiderman]

Set the domain value for rte_pci_addr probing on Windows
to the value of the PCI segment returned by SPDRP_BUSNUMBER.

Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
---
 drivers/bus/pci/windows/pci.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/bus/pci/windows/pci.c b/drivers/bus/pci/windows/pci.c
index 489aa7902a..a40acec609 100644
--- a/drivers/bus/pci/windows/pci.c
+++ b/drivers/bus/pci/windows/pci.c
@@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
 		return -1;
 	}
 
-	addr->domain = 0;
-	addr->bus = bus_num;
+	addr->domain = bus_num >> 8;
+	addr->bus = bus_num & 0xff;
 	addr->devid = dev_and_func >> 16;
 	addr->function = dev_and_func & 0xffff;
 	return 0;
-- 
2.16.1.windows.4

Re: [dpdk-dev] [PATCH] bus/pci: support segment value as address domain on Windows

[Narcisa Ana Maria Vasile]

On Tue, Aug 25, 2020 at 02:43:16PM +0300, Tal Shnaiderman wrote:
> Set the domain value for rte_pci_addr probing on Windows
> to the value of the PCI segment returned by SPDRP_BUSNUMBER.
> 
> Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
> ---
>  drivers/bus/pci/windows/pci.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/bus/pci/windows/pci.c b/drivers/bus/pci/windows/pci.c
> index 489aa7902a..a40acec609 100644
> --- a/drivers/bus/pci/windows/pci.c
> +++ b/drivers/bus/pci/windows/pci.c
> @@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
>  		return -1;
>  	}
>  
> -	addr->domain = 0;
> -	addr->bus = bus_num;
> +	addr->domain = bus_num >> 8;
> +	addr->bus = bus_num & 0xff;
>  	addr->devid = dev_and_func >> 16;
>  	addr->function = dev_and_func & 0xffff;
>  	return 0;
> -- 
Is this needed to avoid collision of devices with the same B:D:F?

Re: [dpdk-dev] [PATCH] bus/pci: support segment value as address domain on Windows

[Tal Shnaiderman]

> Subject: Re: [PATCH] bus/pci: support segment value as address domain on
> Windows
> 
> On Tue, Aug 25, 2020 at 02:43:16PM +0300, Tal Shnaiderman wrote:
> > Set the domain value for rte_pci_addr probing on Windows to the value
> > of the PCI segment returned by SPDRP_BUSNUMBER.
> >
> > Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
> > ---
> >  drivers/bus/pci/windows/pci.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/bus/pci/windows/pci.c
> > b/drivers/bus/pci/windows/pci.c index 489aa7902a..a40acec609 100644
> > --- a/drivers/bus/pci/windows/pci.c
> > +++ b/drivers/bus/pci/windows/pci.c
> > @@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
> >  		return -1;
> >  	}
> >
> > -	addr->domain = 0;
> > -	addr->bus = bus_num;
> > +	addr->domain = bus_num >> 8;
> > +	addr->bus = bus_num & 0xff;
> >  	addr->devid = dev_and_func >> 16;
> >  	addr->function = dev_and_func & 0xffff;
> >  	return 0;
> > --
> Is this needed to avoid collision of devices with the same B:D:F?

Right, it can happen in virtualization setups when several virtual functions can have the same BDF, e.g.: 

PS  > Get-NetAdapterHardwareInfo

Name                           Segment Bus Device Function Slot NumaNode PcieLinkSpeed
----                           ------- --- ------ -------- ---- -------- -------------
Ethernet                                   0   0     10       0                      Unknown
Ethernet 4                       58601   0      2        0             0       Unknown
Ethernet 5                       52956   0      2        0             0       Unknown

DPDK currently can detect either Ethernet 4 or ethernet 5 if only BDF is checked.
Unix uses the Domain value, the equivalent value for Windows is Segment.

Re: [dpdk-dev] [PATCH] bus/pci: support segment value as address domain on Windows

[Menon, Ranjit]

On 9/10/2020 12:30 AM, Tal Shnaiderman wrote:
>> Subject: Re: [PATCH] bus/pci: support segment value as address domain on
>> Windows
>>
>> On Tue, Aug 25, 2020 at 02:43:16PM +0300, Tal Shnaiderman wrote:
>>> Set the domain value for rte_pci_addr probing on Windows to the value
>>> of the PCI segment returned by SPDRP_BUSNUMBER.
>>>
>>> Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
>>> ---
>>>   drivers/bus/pci/windows/pci.c | 4 ++--
>>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/bus/pci/windows/pci.c
>>> b/drivers/bus/pci/windows/pci.c index 489aa7902a..a40acec609 100644
>>> --- a/drivers/bus/pci/windows/pci.c
>>> +++ b/drivers/bus/pci/windows/pci.c
>>> @@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
>>>   		return -1;
>>>   	}
>>>
>>> -	addr->domain = 0;
>>> -	addr->bus = bus_num;
>>> +	addr->domain = bus_num >> 8;
>>> +	addr->bus = bus_num & 0xff;
>>>   	addr->devid = dev_and_func >> 16;
>>>   	addr->function = dev_and_func & 0xffff;
>>>   	return 0;
>>> --
>> Is this needed to avoid collision of devices with the same B:D:F?
> Right, it can happen in virtualization setups when several virtual functions can have the same BDF, e.g.:
>
> PS  > Get-NetAdapterHardwareInfo
>
> Name                           Segment Bus Device Function Slot NumaNode PcieLinkSpeed
> ----                           ------- --- ------ -------- ---- -------- -------------
> Ethernet                                   0   0     10       0                      Unknown
> Ethernet 4                       58601   0      2        0             0       Unknown
> Ethernet 5                       52956   0      2        0             0       Unknown
>
> DPDK currently can detect either Ethernet 4 or ethernet 5 if only BDF is checked.
> Unix uses the Domain value, the equivalent value for Windows is Segment.

Thanks for the explanation, Tal.

I had always been curious how Windows stores the PCIe segment (domain) 
number.

On VMs hosted on Hyper-V, the VF segment numbers are always in the high 
16-bit values.

Is this documented somewhere, or did you find this by experimentation?


ranjit m.

Re: [dpdk-dev] [PATCH] bus/pci: support segment value as address domain on Windows

[Dmitry Kozlyuk]

On Thu, 10 Sep 2020 07:30:39 +0000, Tal Shnaiderman wrote:
> Right, it can happen in virtualization setups when several virtual functions can have the same BDF, e.g.: 
> 
> PS  > Get-NetAdapterHardwareInfo
> 
> Name                           Segment Bus Device Function Slot NumaNode PcieLinkSpeed
> ----                           ------- --- ------ -------- ---- -------- -------------
> Ethernet                                   0   0     10       0                      Unknown
> Ethernet 4                       58601   0      2        0             0       Unknown
> Ethernet 5                       52956   0      2        0             0       Unknown
> 
> DPDK currently can detect either Ethernet 4 or ethernet 5 if only BDF is checked.
> Unix uses the Domain value, the equivalent value for Windows is Segment.

Hi Tal,

I wonder how exactly this setup can be reproduced, i.e. could you
share relevant QEMU options, VMX file or some other config you're using?
Patch idea and code look clear, however, I never managed to build QEMU PCIe
hierarchy to see it working.

Re: [dpdk-dev] [PATCH] bus/pci: support segment value as address domain on Windows

[Tal Shnaiderman]

> Subject: Re: [PATCH] bus/pci: support segment value as address domain on
> Windows
> 
> Thanks for the explanation, Tal.
> 
> I had always been curious how Windows stores the PCIe segment (domain)
> number.
> 
> On VMs hosted on Hyper-V, the VF segment numbers are always in the high
> 16-bit values.
> 
> Is this documented somewhere, or did you find this by experimentation?
> 
Hi Ranjit,

I didn’t find documentation on it.
I found it by experimentation on a Windows VM with several VFs using the same virtual switch, Linux VFs showed the same behavior but they are being detected using the domain value which is different.
> 
> ranjit m.

Re: [dpdk-dev] [PATCH] bus/pci: support segment value as address domain on Windows

[Tal Shnaiderman]

> Subject: Re: [PATCH] bus/pci: support segment value as address domain on
> Windows
> 
> On Thu, 10 Sep 2020 07:30:39 +0000, Tal Shnaiderman wrote:
> > Right, it can happen in virtualization setups when several virtual functions
> can have the same BDF, e.g.:
> >
> > PS  > Get-NetAdapterHardwareInfo
> >
> > Name                           Segment Bus Device Function Slot NumaNode
> PcieLinkSpeed
> > ----                           ------- --- ------ -------- ---- -------- -------------
> > Ethernet                                   0   0     10       0                      Unknown
> > Ethernet 4                       58601   0      2        0             0       Unknown
> > Ethernet 5                       52956   0      2        0             0       Unknown
> >
> > DPDK currently can detect either Ethernet 4 or ethernet 5 if only BDF is
> checked.
> > Unix uses the Domain value, the equivalent value for Windows is Segment.
> 
> Hi Tal,
> 
> I wonder how exactly this setup can be reproduced, i.e. could you share
> relevant QEMU options, VMX file or some other config you're using?
> Patch idea and code look clear, however, I never managed to build QEMU
> PCIe hierarchy to see it working.

Hi Dmitry,

I'm not sure it is relevant to all NICs but for Mellanox you can recreate it on a Hyper-V Windows VM with several SR-IOV VFs using the same virtual switch.

Re: [dpdk-dev] [PATCH] bus/pci: support segment value as address domain on Windows

[Ranjit Menon]

On 8/25/2020 4:43 AM, Tal Shnaiderman wrote:
> Set the domain value for rte_pci_addr probing on Windows
> to the value of the PCI segment returned by SPDRP_BUSNUMBER.
>
> Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
> ---
>   drivers/bus/pci/windows/pci.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/bus/pci/windows/pci.c b/drivers/bus/pci/windows/pci.c
> index 489aa7902a..a40acec609 100644
> --- a/drivers/bus/pci/windows/pci.c
> +++ b/drivers/bus/pci/windows/pci.c
> @@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
>   		return -1;
>   	}
>   
> -	addr->domain = 0;
> -	addr->bus = bus_num;
> +	addr->domain = bus_num >> 8;
> +	addr->bus = bus_num & 0xff;
>   	addr->devid = dev_and_func >> 16;
>   	addr->function = dev_and_func & 0xffff;
>   	return 0;


Acked-by: Ranjit Menon <ranjit.menon@intel.com>

Re: [dpdk-dev] [PATCH] bus/pci: support segment value as address domain on Windows

[Thomas Monjalon]

16/09/2020 20:32, Ranjit Menon:
> On 8/25/2020 4:43 AM, Tal Shnaiderman wrote:
> > Set the domain value for rte_pci_addr probing on Windows
> > to the value of the PCI segment returned by SPDRP_BUSNUMBER.
> >
> > Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
> > ---
> > --- a/drivers/bus/pci/windows/pci.c
> > +++ b/drivers/bus/pci/windows/pci.c
> > @@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
> > -	addr->domain = 0;
> > -	addr->bus = bus_num;
> > +	addr->domain = bus_num >> 8;
> > +	addr->bus = bus_num & 0xff;
> >   	addr->devid = dev_and_func >> 16;
> >   	addr->function = dev_and_func & 0xffff;
> 
> 
> Acked-by: Ranjit Menon <ranjit.menon@intel.com>

I was waiting for feedback from Microsoft, but never happened, so
Applied, thanks

Microsoft, we need a better support for porting DPDK on Windows.