Regarding port numbers assigned by DPDK for PCI devices

Regarding port numbers assigned by DPDK for PCI devices

[Prashant Upadhyaya]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

Hi,

I have 4 ethernet interfaces available as PCI devices.
The PCI addresses are known.
When I start my DPDK application, it starts up properly and assigns the
port numbers to them as 0, 1, 2, 3 expectedly.

However, is there a way I can force that a particular PCI address should be
identified as port 0 by DPDK, another one as port 1 and so forth ?
Does passing the arguments like -a <address1> -a <address2> to rte_eal_init
ensure that, or is there any other way, or is there no way ?

I am on 24.03

Regards
-Prashant

[-- Attachment #2: Type: text/html, Size: 708 bytes --]

Re: Regarding port numbers assigned by DPDK for PCI devices

[Dmitry Kozlyuk]

2024-07-29 22:18 (UTC+0530), Prashant Upadhyaya:
> Hi,
> 
> I have 4 ethernet interfaces available as PCI devices.
> The PCI addresses are known.
> When I start my DPDK application, it starts up properly and assigns the
> port numbers to them as 0, 1, 2, 3 expectedly.
> 
> However, is there a way I can force that a particular PCI address should be
> identified as port 0 by DPDK, another one as port 1 and so forth ?
> Does passing the arguments like -a <address1> -a <address2> to rte_eal_init
> ensure that, or is there any other way, or is there no way ?
> 
> I am on 24.03
> 
> Regards
> -Prashant

Hi,

Why do you need specific port numbers assigned to devices?

If you're going to use devices for different purposes,
you'd better have an application-level configuration
to specify PCI addresses to use for each purpose.
Iterate devices to match the address and find the port number.

It is also possible to disable automatic probing with "-a 0:0.0",
then to use rte_dev_probe() + rte_dev_event_callback_register()
to add devices and to get their port numbers.
However, this API, strictly speaking, does not guarantee
that the numbers will be assigned sequentially.
One advantage of using hot-plug is that you can build devargs
from within the application (or from configuration).

Refer to "rte_dev.h" in any case.
Multiple "-a" don't work the way you've described.

Re: Regarding port numbers assigned by DPDK for PCI devices

[Prashant Upadhyaya]

[-- Attachment #1: Type: text/plain, Size: 2088 bytes --]

On Mon, 29 Jul 2024 at 23:13, Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
wrote:

> 2024-07-29 22:18 (UTC+0530), Prashant Upadhyaya:
> > Hi,
> >
> > I have 4 ethernet interfaces available as PCI devices.
> > The PCI addresses are known.
> > When I start my DPDK application, it starts up properly and assigns the
> > port numbers to them as 0, 1, 2, 3 expectedly.
> >
> > However, is there a way I can force that a particular PCI address should
> be
> > identified as port 0 by DPDK, another one as port 1 and so forth ?
> > Does passing the arguments like -a <address1> -a <address2> to
> rte_eal_init
> > ensure that, or is there any other way, or is there no way ?
> >
> > I am on 24.03
> >
> > Regards
> > -Prashant
>
> Hi,
>
> Why do you need specific port numbers assigned to devices?
>
> If you're going to use devices for different purposes,
> you'd better have an application-level configuration
> to specify PCI addresses to use for each purpose.
> Iterate devices to match the address and find the port number.
>
> It is also possible to disable automatic probing with "-a 0:0.0",
> then to use rte_dev_probe() + rte_dev_event_callback_register()
> to add devices and to get their port numbers.
> However, this API, strictly speaking, does not guarantee
> that the numbers will be assigned sequentially.
> One advantage of using hot-plug is that you can build devargs
> from within the application (or from configuration).
>
> Refer to "rte_dev.h" in any case.
> Multiple "-a" don't work the way you've described.
>


Thanks Dmitry. Ok, so if I have the port number with me, and I know it
corresponds to a PCI device, how do I find out the PCI address of this
device corresponding to this port number. I believe I can
do rte_eth_dev_info_get to get the struct rte_eth_dev_info and from there
the rte_device, but what after that ? I saw some references
to RTE_DEV_TO_PCI but that macro isn't available for compilation after DPDK
is installed as it is an internal header file and thus not a macro for
application usage and wouldn't compile at application level.

Regards
-Prashant

[-- Attachment #2: Type: text/html, Size: 2777 bytes --]

Re: Regarding port numbers assigned by DPDK for PCI devices

[Bruce Richardson]

On Tue, Jul 30, 2024 at 11:06:56AM +0530, Prashant Upadhyaya wrote:
>    On Mon, 29 Jul 2024 at 23:13, Dmitry Kozlyuk
>    <[1]dmitry.kozliuk@gmail.com> wrote:
> 
>      2024-07-29 22:18 (UTC+0530), Prashant Upadhyaya:
>      > Hi,
>      >
>      > I have 4 ethernet interfaces available as PCI devices.
>      > The PCI addresses are known.
>      > When I start my DPDK application, it starts up properly and
>      assigns the
>      > port numbers to them as 0, 1, 2, 3 expectedly.
>      >
>      > However, is there a way I can force that a particular PCI address
>      should be
>      > identified as port 0 by DPDK, another one as port 1 and so forth ?
>      > Does passing the arguments like -a <address1> -a <address2> to
>      rte_eal_init
>      > ensure that, or is there any other way, or is there no way ?
>      >
>      > I am on 24.03
>      >
>      > Regards
>      > -Prashant
>      Hi,
>      Why do you need specific port numbers assigned to devices?
>      If you're going to use devices for different purposes,
>      you'd better have an application-level configuration
>      to specify PCI addresses to use for each purpose.
>      Iterate devices to match the address and find the port number.
>      It is also possible to disable automatic probing with "-a 0:0.0",
>      then to use rte_dev_probe() + rte_dev_event_callback_register()
>      to add devices and to get their port numbers.
>      However, this API, strictly speaking, does not guarantee
>      that the numbers will be assigned sequentially.
>      One advantage of using hot-plug is that you can build devargs
>      from within the application (or from configuration).
>      Refer to "rte_dev.h" in any case.
>      Multiple "-a" don't work the way you've described.
> 
>    Thanks Dmitry. Ok, so if I have the port number with me, and I know it
>    corresponds to a PCI device, how do I find out the PCI address of this
>    device corresponding to this port number. I believe I can
>    do rte_eth_dev_info_get to get the struct rte_eth_dev_info and from
>    there the rte_device, but what after that ? I saw some references
>    to RTE_DEV_TO_PCI but that macro isn't available for compilation after
>    DPDK is installed as it is an internal header file and thus not a macro
>    for application usage and wouldn't compile at application level.
>    Regards
>    -Prashant
> 
The PCI device id is used as the device name in that case so rte_dev_name
should get you what you want.

I'd also +1 the suggestion of having your app hotplug in the devices
post-init, if you want specific devices to have specific ids. Although it
is not guaranteed, DPDK does currently assign the ids sequentially. I've
used this approach myself in the past in some test apps where I wanted
ports in a particular sequence.

/Bruce

Re: Regarding port numbers assigned by DPDK for PCI devices

[Ferruh Yigit]

On 7/30/2024 6:36 AM, Prashant Upadhyaya wrote:
> 
> 
> 
> On Mon, 29 Jul 2024 at 23:13, Dmitry Kozlyuk <dmitry.kozliuk@gmail.com
> <mailto:dmitry.kozliuk@gmail.com>> wrote:
> 
>     2024-07-29 22:18 (UTC+0530), Prashant Upadhyaya:
>     > Hi,
>     >
>     > I have 4 ethernet interfaces available as PCI devices.
>     > The PCI addresses are known.
>     > When I start my DPDK application, it starts up properly and
>     assigns the
>     > port numbers to them as 0, 1, 2, 3 expectedly.
>     >
>     > However, is there a way I can force that a particular PCI address
>     should be
>     > identified as port 0 by DPDK, another one as port 1 and so forth ?
>     > Does passing the arguments like -a <address1> -a <address2> to
>     rte_eal_init
>     > ensure that, or is there any other way, or is there no way ?
>     >
>     > I am on 24.03
>     >
>     > Regards
>     > -Prashant
> 
>     Hi,
> 
>     Why do you need specific port numbers assigned to devices?
> 
>     If you're going to use devices for different purposes,
>     you'd better have an application-level configuration
>     to specify PCI addresses to use for each purpose.
>     Iterate devices to match the address and find the port number.
> 
>     It is also possible to disable automatic probing with "-a 0:0.0",
>     then to use rte_dev_probe() + rte_dev_event_callback_register()
>     to add devices and to get their port numbers.
>     However, this API, strictly speaking, does not guarantee
>     that the numbers will be assigned sequentially.
>     One advantage of using hot-plug is that you can build devargs
>     from within the application (or from configuration).
> 
>     Refer to "rte_dev.h" in any case.
>     Multiple "-a" don't work the way you've described.
> 
> 
> 
> Thanks Dmitry. Ok, so if I have the port number with me, and I know it
> corresponds to a PCI device, how do I find out the PCI address of this
> device corresponding to this port number. I believe I can
> do rte_eth_dev_info_get to get the struct rte_eth_dev_info and from
> there the rte_device, but what after that ? I saw some references
> to RTE_DEV_TO_PCI but that macro isn't available for compilation after
> DPDK is installed as it is an internal header file and thus not a macro
> for application usage and wouldn't compile at application level.
> 
> 

Hi Prashant,

For PCI bus, most of the times ethdev device name is PCI ID, so you can
use 'rte_eth_dev_get_name_by_port(port_id, name)' API to get PCI ID for
port.

But there are cases this is not true, like single PCI ID creates
multiple ethdev etc... for this case, PCI ID can be get from device name:
rte_eth_dev_info_get(port_id, *dev_info)
name = rte_dev_name(dev_info->device)

Re: Regarding port numbers assigned by DPDK for PCI devices

[Prashant Upadhyaya]

[-- Attachment #1: Type: text/plain, Size: 2981 bytes --]

On Wed, 31 Jul 2024 at 14:35, Ferruh Yigit <ferruh.yigit@amd.com> wrote:

> On 7/30/2024 6:36 AM, Prashant Upadhyaya wrote:
> >
> >
> >
> > On Mon, 29 Jul 2024 at 23:13, Dmitry Kozlyuk <dmitry.kozliuk@gmail.com
> > <mailto:dmitry.kozliuk@gmail.com>> wrote:
> >
> >     2024-07-29 22:18 (UTC+0530), Prashant Upadhyaya:
> >     > Hi,
> >     >
> >     > I have 4 ethernet interfaces available as PCI devices.
> >     > The PCI addresses are known.
> >     > When I start my DPDK application, it starts up properly and
> >     assigns the
> >     > port numbers to them as 0, 1, 2, 3 expectedly.
> >     >
> >     > However, is there a way I can force that a particular PCI address
> >     should be
> >     > identified as port 0 by DPDK, another one as port 1 and so forth ?
> >     > Does passing the arguments like -a <address1> -a <address2> to
> >     rte_eal_init
> >     > ensure that, or is there any other way, or is there no way ?
> >     >
> >     > I am on 24.03
> >     >
> >     > Regards
> >     > -Prashant
> >
> >     Hi,
> >
> >     Why do you need specific port numbers assigned to devices?
> >
> >     If you're going to use devices for different purposes,
> >     you'd better have an application-level configuration
> >     to specify PCI addresses to use for each purpose.
> >     Iterate devices to match the address and find the port number.
> >
> >     It is also possible to disable automatic probing with "-a 0:0.0",
> >     then to use rte_dev_probe() + rte_dev_event_callback_register()
> >     to add devices and to get their port numbers.
> >     However, this API, strictly speaking, does not guarantee
> >     that the numbers will be assigned sequentially.
> >     One advantage of using hot-plug is that you can build devargs
> >     from within the application (or from configuration).
> >
> >     Refer to "rte_dev.h" in any case.
> >     Multiple "-a" don't work the way you've described.
> >
> >
> >
> > Thanks Dmitry. Ok, so if I have the port number with me, and I know it
> > corresponds to a PCI device, how do I find out the PCI address of this
> > device corresponding to this port number. I believe I can
> > do rte_eth_dev_info_get to get the struct rte_eth_dev_info and from
> > there the rte_device, but what after that ? I saw some references
> > to RTE_DEV_TO_PCI but that macro isn't available for compilation after
> > DPDK is installed as it is an internal header file and thus not a macro
> > for application usage and wouldn't compile at application level.
> >
> >
>
> Hi Prashant,
>
> For PCI bus, most of the times ethdev device name is PCI ID, so you can
> use 'rte_eth_dev_get_name_by_port(port_id, name)' API to get PCI ID for
> port.
>
> But there are cases this is not true, like single PCI ID creates
> multiple ethdev etc... for this case, PCI ID can be get from device name:
> rte_eth_dev_info_get(port_id, *dev_info)
> name = rte_dev_name(dev_info->device)
>
>
>
Thanks Ferruh, this was helpful.

Regards
-Prashant

[-- Attachment #2: Type: text/html, Size: 4140 bytes --]

Re: Regarding port numbers assigned by DPDK for PCI devices

[Prashant Upadhyaya]

[-- Attachment #1: Type: text/plain, Size: 3073 bytes --]

On Tue, 30 Jul 2024 at 13:45, Bruce Richardson <bruce.richardson@intel.com>
wrote:

> On Tue, Jul 30, 2024 at 11:06:56AM +0530, Prashant Upadhyaya wrote:
> >    On Mon, 29 Jul 2024 at 23:13, Dmitry Kozlyuk
> >    <[1]dmitry.kozliuk@gmail.com> wrote:
> >
> >      2024-07-29 22:18 (UTC+0530), Prashant Upadhyaya:
> >      > Hi,
> >      >
> >      > I have 4 ethernet interfaces available as PCI devices.
> >      > The PCI addresses are known.
> >      > When I start my DPDK application, it starts up properly and
> >      assigns the
> >      > port numbers to them as 0, 1, 2, 3 expectedly.
> >      >
> >      > However, is there a way I can force that a particular PCI address
> >      should be
> >      > identified as port 0 by DPDK, another one as port 1 and so forth ?
> >      > Does passing the arguments like -a <address1> -a <address2> to
> >      rte_eal_init
> >      > ensure that, or is there any other way, or is there no way ?
> >      >
> >      > I am on 24.03
> >      >
> >      > Regards
> >      > -Prashant
> >      Hi,
> >      Why do you need specific port numbers assigned to devices?
> >      If you're going to use devices for different purposes,
> >      you'd better have an application-level configuration
> >      to specify PCI addresses to use for each purpose.
> >      Iterate devices to match the address and find the port number.
> >      It is also possible to disable automatic probing with "-a 0:0.0",
> >      then to use rte_dev_probe() + rte_dev_event_callback_register()
> >      to add devices and to get their port numbers.
> >      However, this API, strictly speaking, does not guarantee
> >      that the numbers will be assigned sequentially.
> >      One advantage of using hot-plug is that you can build devargs
> >      from within the application (or from configuration).
> >      Refer to "rte_dev.h" in any case.
> >      Multiple "-a" don't work the way you've described.
> >
> >    Thanks Dmitry. Ok, so if I have the port number with me, and I know it
> >    corresponds to a PCI device, how do I find out the PCI address of this
> >    device corresponding to this port number. I believe I can
> >    do rte_eth_dev_info_get to get the struct rte_eth_dev_info and from
> >    there the rte_device, but what after that ? I saw some references
> >    to RTE_DEV_TO_PCI but that macro isn't available for compilation after
> >    DPDK is installed as it is an internal header file and thus not a
> macro
> >    for application usage and wouldn't compile at application level.
> >    Regards
> >    -Prashant
> >
> The PCI device id is used as the device name in that case so rte_dev_name
> should get you what you want.
>
> I'd also +1 the suggestion of having your app hotplug in the devices
> post-init, if you want specific devices to have specific ids. Although it
> is not guaranteed, DPDK does currently assign the ids sequentially. I've
> used this approach myself in the past in some test apps where I wanted
> ports in a particular sequence.
>
> /Bruce
>

Thanks Bruce, makes sense.

Regards
-Prashant

[-- Attachment #2: Type: text/html, Size: 4108 bytes --]