Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[Loftus, Ciara]

> Hi,
> 
> I have compiled and built ovs-dpdk using DPDK v17.08 and OVS v2.8.0. The
> NIC that I am using is Mellanox ConnectX-3 Pro, which is a dual port 10G
> NIC. The problem with this NIC is that it provides only one PCI address for
> both the 10G ports.
> 
> So when I am trying to add the two DPDK ports to my br0 bridge
> 
> # ovs-vsctl --no-wait add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> options:dpdk-devargs=0002:01:00.0
> 
> # ovs-vsctl --no-wait add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk
> options:dpdk-devargs=0002:01:00.0
> 
> The port dpdk1 is added successfully and able to transfer data, but adding
> dpdk0 to br0 fails:
> 
> 2017-09-06T14:19:20Z|00045|netdev_dpdk|INFO|Port 0: e4:1d:2d:4f:78:60
> 2017-09-06T14:19:20Z|00046|bridge|INFO|bridge br0: added interface dpdk1
> on
> port 1
> 2017-09-06T14:19:20Z|00047|bridge|INFO|bridge br0: added interface br0
> on
> port 65534
> 2017-09-06T14:19:20Z|00048|dpif_netlink|WARN|Generic Netlink family
> 'ovs_datapath' does not exist. The Open vSwitch kernel module is probably
> not loaded.
> 2017-09-06T14:19:20Z|00049|netdev_dpdk|WARN|'dpdk0' is trying to use
> device
> '0002:01:00.0' which is already in use by 'dpdk1'
> 2017-09-06T14:19:20Z|00050|netdev|WARN|dpdk0: could not set
> configuration
> (Address already in use)
> 2017-09-06T14:19:20Z|00051|bridge|INFO|bridge br0: using datapath ID
> 0000e41d2d4f7860
> 
> 
> With OVS v2.6.1 I never had this problem as dpdk-devargs was not
> mandatory
> and just specifying port name was enough to add that port to bridge.
> 
> Is there a way to add port both ports to bridge ?

It seems the DPDK function rte_eth_dev_get_port_by_name() will always return the port ID of the first port on your NIC, when you specify the single PCI address and that's where the problem is. There doesn't seem to be a way currently to indicate to the calling application that in fact two (or more) port IDs are associated with the one PCI address.

I am cc-ing DPDK users mailing list for hopefully some input. Are there any plans for the rte_eth_dev_get_port_by_name function to be compatible with NICs with multiple ports under the same PCI address?

Thanks,
Ciara

> 
> Thanks,
> Devendra
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[Kevin Traynor]

On 09/08/2017 10:56 AM, Loftus, Ciara wrote:
>> Hi,
>>
>> I have compiled and built ovs-dpdk using DPDK v17.08 and OVS v2.8.0. The
>> NIC that I am using is Mellanox ConnectX-3 Pro, which is a dual port 10G
>> NIC. The problem with this NIC is that it provides only one PCI address for
>> both the 10G ports.
>>
>> So when I am trying to add the two DPDK ports to my br0 bridge
>>
>> # ovs-vsctl --no-wait add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
>> options:dpdk-devargs=0002:01:00.0
>>
>> # ovs-vsctl --no-wait add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk
>> options:dpdk-devargs=0002:01:00.0
>>
>> The port dpdk1 is added successfully and able to transfer data, but adding
>> dpdk0 to br0 fails:
>>
>> 2017-09-06T14:19:20Z|00045|netdev_dpdk|INFO|Port 0: e4:1d:2d:4f:78:60
>> 2017-09-06T14:19:20Z|00046|bridge|INFO|bridge br0: added interface dpdk1
>> on
>> port 1
>> 2017-09-06T14:19:20Z|00047|bridge|INFO|bridge br0: added interface br0
>> on
>> port 65534
>> 2017-09-06T14:19:20Z|00048|dpif_netlink|WARN|Generic Netlink family
>> 'ovs_datapath' does not exist. The Open vSwitch kernel module is probably
>> not loaded.
>> 2017-09-06T14:19:20Z|00049|netdev_dpdk|WARN|'dpdk0' is trying to use
>> device
>> '0002:01:00.0' which is already in use by 'dpdk1'
>> 2017-09-06T14:19:20Z|00050|netdev|WARN|dpdk0: could not set
>> configuration
>> (Address already in use)
>> 2017-09-06T14:19:20Z|00051|bridge|INFO|bridge br0: using datapath ID
>> 0000e41d2d4f7860
>>
>>
>> With OVS v2.6.1 I never had this problem as dpdk-devargs was not
>> mandatory
>> and just specifying port name was enough to add that port to bridge.
>>
>> Is there a way to add port both ports to bridge ?
> 
> It seems the DPDK function rte_eth_dev_get_port_by_name() will always return the port ID of the first port on your NIC, when you specify the single PCI address and that's where the problem is. There doesn't seem to be a way currently to indicate to the calling application that in fact two (or more) port IDs are associated with the one PCI address.
> 
> I am cc-ing DPDK users mailing list for hopefully some input. Are there any plans for the rte_eth_dev_get_port_by_name function to be compatible with NICs with multiple ports under the same PCI address?
> 

Hi Adrien/Nelio,

Is this something you can answer? We're wondering how to handle this in
OVS and whether a temporary or long term solution is needed.

The original thread started here:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-September/338418.html

thanks,
Kevin.

> Thanks,
> Ciara
> 
>>
>> Thanks,
>> Devendra
>> _______________________________________________
>> dev mailing list
>> dev@openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[Thomas Monjalon]

20/09/2017 19:33, Kevin Traynor:
> On 09/08/2017 10:56 AM, Loftus, Ciara wrote:
> >> Hi,
> >>
> >> I have compiled and built ovs-dpdk using DPDK v17.08 and OVS v2.8.0. The
> >> NIC that I am using is Mellanox ConnectX-3 Pro, which is a dual port 10G
> >> NIC. The problem with this NIC is that it provides only one PCI address for
> >> both the 10G ports.
> >>
> >> So when I am trying to add the two DPDK ports to my br0 bridge
> >>
> >> # ovs-vsctl --no-wait add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> >> options:dpdk-devargs=0002:01:00.0
> >>
> >> # ovs-vsctl --no-wait add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk
> >> options:dpdk-devargs=0002:01:00.0
> >>
> >> The port dpdk1 is added successfully and able to transfer data, but adding
> >> dpdk0 to br0 fails:
[...]
> >> With OVS v2.6.1 I never had this problem as dpdk-devargs was not
> >> mandatory
> >> and just specifying port name was enough to add that port to bridge.
> >>
> >> Is there a way to add port both ports to bridge ?
> > 
> > It seems the DPDK function rte_eth_dev_get_port_by_name() will always return the port ID of the first port on your NIC, when you specify the single PCI address and that's where the problem is. There doesn't seem to be a way currently to indicate to the calling application that in fact two (or more) port IDs are associated with the one PCI address.

We have two ports (with the same PCI address) so we should have
two different names.
Where the names passed to rte_eth_dev_get_port_by_name() come from?
It is the user parameter from options:dpdk-devargs=0002:01:00.0, right?

> > I am cc-ing DPDK users mailing list for hopefully some input. Are there any plans for the rte_eth_dev_get_port_by_name function to be compatible with NICs with multiple ports under the same PCI address?

We cannot return two different ports for the same name.
There are two issues here:
	- the input should not be the PCI address
	- the ethdev function should look at ethdev name, not rte_device one

The idea is that we have only one rte_device object and we instantiate
two rte_eth_dev ports.
An ethdev port can be identified with its id (a number) or its unique name.
Unfortunately, the user cannot guess the port id or the name set by the PMD.

> Hi Adrien/Nelio,
> 
> Is this something you can answer? We're wondering how to handle this in
> OVS and whether a temporary or long term solution is needed.

I suggest to rely on ethdev name.
You will need to show to the user the mapping between the bus information
(PCI id here) and the device names.

Another alternative is to add a new function returning all ethdev ports
associated to a given rte_device resource.
So you would get two ports and you could pick one on the first "add-port",
and the other one for the second "add-port" command.
It means the user would be forced to add them in the right order if he
wants a reproducible result.

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[Loftus, Ciara]

> 20/09/2017 19:33, Kevin Traynor:
> > On 09/08/2017 10:56 AM, Loftus, Ciara wrote:
> > >> Hi,
> > >>
> > >> I have compiled and built ovs-dpdk using DPDK v17.08 and OVS v2.8.0.
> The
> > >> NIC that I am using is Mellanox ConnectX-3 Pro, which is a dual port 10G
> > >> NIC. The problem with this NIC is that it provides only one PCI address
> for
> > >> both the 10G ports.
> > >>
> > >> So when I am trying to add the two DPDK ports to my br0 bridge
> > >>
> > >> # ovs-vsctl --no-wait add-port br0 dpdk0 -- set Interface dpdk0
> type=dpdk
> > >> options:dpdk-devargs=0002:01:00.0
> > >>
> > >> # ovs-vsctl --no-wait add-port br0 dpdk1 -- set Interface dpdk1
> type=dpdk
> > >> options:dpdk-devargs=0002:01:00.0
> > >>
> > >> The port dpdk1 is added successfully and able to transfer data, but
> adding
> > >> dpdk0 to br0 fails:
> [...]
> > >> With OVS v2.6.1 I never had this problem as dpdk-devargs was not
> > >> mandatory
> > >> and just specifying port name was enough to add that port to bridge.
> > >>
> > >> Is there a way to add port both ports to bridge ?
> > >
> > > It seems the DPDK function rte_eth_dev_get_port_by_name() will
> always return the port ID of the first port on your NIC, when you specify the
> single PCI address and that's where the problem is. There doesn't seem to be
> a way currently to indicate to the calling application that in fact two (or more)
> port IDs are associated with the one PCI address.
> 
> We have two ports (with the same PCI address) so we should have
> two different names.
> Where the names passed to rte_eth_dev_get_port_by_name() come from?
> It is the user parameter from options:dpdk-devargs=0002:01:00.0, right?

Yes, we're using the PCI address specified by the user in dpdk-devargs.

> 
> > > I am cc-ing DPDK users mailing list for hopefully some input. Are there any
> plans for the rte_eth_dev_get_port_by_name function to be compatible
> with NICs with multiple ports under the same PCI address?
> 
> We cannot return two different ports for the same name.
> There are two issues here:
> 	- the input should not be the PCI address
> 	- the ethdev function should look at ethdev name, not rte_device
> one

This would require the user having to "guess" the DPDK ethdev name which is something we'd like to avoid.
We had the same problem using DPDK port IDs and decided not to use them anymore, and use the PCI instead as it took the guesswork out.
Ethdev names and port IDs can change between tests, unlike the PCI address which tends to remain constant for a device.

> 
> The idea is that we have only one rte_device object and we instantiate
> two rte_eth_dev ports.
> An ethdev port can be identified with its id (a number) or its unique name.
> Unfortunately, the user cannot guess the port id or the name set by the
> PMD.

Exactly. Thanks for clarifying what's going on under the hood.

Ciara

> 
> > Hi Adrien/Nelio,
> >
> > Is this something you can answer? We're wondering how to handle this in
> > OVS and whether a temporary or long term solution is needed.
> 
> I suggest to rely on ethdev name.
> You will need to show to the user the mapping between the bus information
> (PCI id here) and the device names.
> 
> Another alternative is to add a new function returning all ethdev ports
> associated to a given rte_device resource.
> So you would get two ports and you could pick one on the first "add-port",
> and the other one for the second "add-port" command.
> It means the user would be forced to add them in the right order if he
> wants a reproducible result.

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[Thomas Monjalon]

21/09/2017 10:04, Loftus, Ciara:
> > 20/09/2017 19:33, Kevin Traynor:
> > > On 09/08/2017 10:56 AM, Loftus, Ciara wrote:
> > > > It seems the DPDK function rte_eth_dev_get_port_by_name() will
> > > > always return the port ID of the first port on your NIC,
> > > > when you specify the single PCI address and that's where the
> > > > problem is. There doesn't seem to be a way currently to
> > > > indicate to the calling application that in fact two
> > > > (or more) port IDs are associated with the one PCI address.
> > 
> > We have two ports (with the same PCI address) so we should have
> > two different names.
> > Where the names passed to rte_eth_dev_get_port_by_name() come from?
> > It is the user parameter from options:dpdk-devargs=0002:01:00.0, right?
> 
> Yes, we're using the PCI address specified by the user in dpdk-devargs.
> 
> > > > I am cc-ing DPDK users mailing list for hopefully some input.
> > > > Are there any plans for the rte_eth_dev_get_port_by_name function
> > > > to be compatible with NICs with multiple ports under the same PCI address?
> > 
> > We cannot return two different ports for the same name.
> > There are two issues here:
> > 	- the input should not be the PCI address
> > 	- the ethdev function should look at ethdev name, not rte_device
> > one
> 
> This would require the user having to "guess" the DPDK ethdev name
> which is something we'd like to avoid.

Yes, but you can provide a way to list the ports with their names
and characteristics.

> We had the same problem using DPDK port IDs and decided not to use
> them anymore, and use the PCI instead as it took the guesswork out.
> Ethdev names and port IDs can change between tests, unlike the PCI
> address which tends to remain constant for a device.

We can add a requirement on ethdev names and make sure they remain
constant for a given port.

> > The idea is that we have only one rte_device object and we instantiate
> > two rte_eth_dev ports.
> > An ethdev port can be identified with its id (a number) or its unique name.
> > Unfortunately, the user cannot guess the port id or the name set by the
> > PMD.
> 
> Exactly. Thanks for clarifying what's going on under the hood.
> 
> Ciara
> 
> > 
> > > Hi Adrien/Nelio,
> > >
> > > Is this something you can answer? We're wondering how to handle this in
> > > OVS and whether a temporary or long term solution is needed.
> > 
> > I suggest to rely on ethdev name.
> > You will need to show to the user the mapping between the bus information
> > (PCI id here) and the device names.
> > 
> > Another alternative is to add a new function returning all ethdev ports
> > associated to a given rte_device resource.
> > So you would get two ports and you could pick one on the first "add-port",
> > and the other one for the second "add-port" command.
> > It means the user would be forced to add them in the right order if he
> > wants a reproducible result.

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[Loftus, Ciara]

> 21/09/2017 10:04, Loftus, Ciara:
> > > 20/09/2017 19:33, Kevin Traynor:
> > > > On 09/08/2017 10:56 AM, Loftus, Ciara wrote:
> > > > > It seems the DPDK function rte_eth_dev_get_port_by_name() will
> > > > > always return the port ID of the first port on your NIC,
> > > > > when you specify the single PCI address and that's where the
> > > > > problem is. There doesn't seem to be a way currently to
> > > > > indicate to the calling application that in fact two
> > > > > (or more) port IDs are associated with the one PCI address.
> > >
> > > We have two ports (with the same PCI address) so we should have
> > > two different names.
> > > Where the names passed to rte_eth_dev_get_port_by_name() come
> from?
> > > It is the user parameter from options:dpdk-devargs=0002:01:00.0, right?
> >
> > Yes, we're using the PCI address specified by the user in dpdk-devargs.
> >
> > > > > I am cc-ing DPDK users mailing list for hopefully some input.
> > > > > Are there any plans for the rte_eth_dev_get_port_by_name
> function
> > > > > to be compatible with NICs with multiple ports under the same PCI
> address?
> > >
> > > We cannot return two different ports for the same name.
> > > There are two issues here:
> > > 	- the input should not be the PCI address
> > > 	- the ethdev function should look at ethdev name, not rte_device
> > > one
> >
> > This would require the user having to "guess" the DPDK ethdev name
> > which is something we'd like to avoid.
> 
> Yes, but you can provide a way to list the ports with their names
> and characteristics.

Ok, I see. Maybe something like this could be considered:

port A = dpdk-devargs=xx:yy:zz 0
port B = dpdk-devargs=xx:yy:zz 1

If we detect a value after the PCI address we iterate through the rte_eth_dev_info (http://dpdk.org/doc/api/structrte__eth__dev__info.html) for all valid port IDs and assign port A to the first ethdev encountered with the provided PCI address, and port B to the second, etc.

If we don't detect a value, then we operate as normal. Thoughts?

Thanks,
Ciara

> 
> > We had the same problem using DPDK port IDs and decided not to use
> > them anymore, and use the PCI instead as it took the guesswork out.
> > Ethdev names and port IDs can change between tests, unlike the PCI
> > address which tends to remain constant for a device.
> 
> We can add a requirement on ethdev names and make sure they remain
> constant for a given port.
> 
> > > The idea is that we have only one rte_device object and we instantiate
> > > two rte_eth_dev ports.
> > > An ethdev port can be identified with its id (a number) or its unique
> name.
> > > Unfortunately, the user cannot guess the port id or the name set by the
> > > PMD.
> >
> > Exactly. Thanks for clarifying what's going on under the hood.
> >
> > Ciara
> >
> > >
> > > > Hi Adrien/Nelio,
> > > >
> > > > Is this something you can answer? We're wondering how to handle this
> in
> > > > OVS and whether a temporary or long term solution is needed.
> > >
> > > I suggest to rely on ethdev name.
> > > You will need to show to the user the mapping between the bus
> information
> > > (PCI id here) and the device names.
> > >
> > > Another alternative is to add a new function returning all ethdev ports
> > > associated to a given rte_device resource.
> > > So you would get two ports and you could pick one on the first "add-
> port",
> > > and the other one for the second "add-port" command.
> > > It means the user would be forced to add them in the right order if he
> > > wants a reproducible result.

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[Adrien Mazarguil]

On Wed, Sep 20, 2017 at 06:33:43PM +0100, Kevin Traynor wrote:
> On 09/08/2017 10:56 AM, Loftus, Ciara wrote:
> >> Hi,
> >>
> >> I have compiled and built ovs-dpdk using DPDK v17.08 and OVS v2.8.0. The
> >> NIC that I am using is Mellanox ConnectX-3 Pro, which is a dual port 10G
> >> NIC. The problem with this NIC is that it provides only one PCI address for
> >> both the 10G ports.
> >>
> >> So when I am trying to add the two DPDK ports to my br0 bridge
> >>
> >> # ovs-vsctl --no-wait add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> >> options:dpdk-devargs=0002:01:00.0
> >>
> >> # ovs-vsctl --no-wait add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk
> >> options:dpdk-devargs=0002:01:00.0
> >>
> >> The port dpdk1 is added successfully and able to transfer data, but adding
> >> dpdk0 to br0 fails:
> >>
> >> 2017-09-06T14:19:20Z|00045|netdev_dpdk|INFO|Port 0: e4:1d:2d:4f:78:60
> >> 2017-09-06T14:19:20Z|00046|bridge|INFO|bridge br0: added interface dpdk1
> >> on
> >> port 1
> >> 2017-09-06T14:19:20Z|00047|bridge|INFO|bridge br0: added interface br0
> >> on
> >> port 65534
> >> 2017-09-06T14:19:20Z|00048|dpif_netlink|WARN|Generic Netlink family
> >> 'ovs_datapath' does not exist. The Open vSwitch kernel module is probably
> >> not loaded.
> >> 2017-09-06T14:19:20Z|00049|netdev_dpdk|WARN|'dpdk0' is trying to use
> >> device
> >> '0002:01:00.0' which is already in use by 'dpdk1'
> >> 2017-09-06T14:19:20Z|00050|netdev|WARN|dpdk0: could not set
> >> configuration
> >> (Address already in use)
> >> 2017-09-06T14:19:20Z|00051|bridge|INFO|bridge br0: using datapath ID
> >> 0000e41d2d4f7860
> >>
> >>
> >> With OVS v2.6.1 I never had this problem as dpdk-devargs was not
> >> mandatory
> >> and just specifying port name was enough to add that port to bridge.
> >>
> >> Is there a way to add port both ports to bridge ?
> > 
> > It seems the DPDK function rte_eth_dev_get_port_by_name() will always return the port ID of the first port on your NIC, when you specify the single PCI address and that's where the problem is. There doesn't seem to be a way currently to indicate to the calling application that in fact two (or more) port IDs are associated with the one PCI address.
> > 
> > I am cc-ing DPDK users mailing list for hopefully some input. Are there any plans for the rte_eth_dev_get_port_by_name function to be compatible with NICs with multiple ports under the same PCI address?
> > 
> 
> Hi Adrien/Nelio,
> 
> Is this something you can answer? We're wondering how to handle this in
> OVS and whether a temporary or long term solution is needed.
> 
> The original thread started here:
> https://mail.openvswitch.org/pipermail/ovs-dev/2017-September/338418.html

Hi,

I'm late to this thread and not too familiar with openvswitch's internals,
however after reading this thread here are some thoughts.

About rte_eth_dev_get_port_by_name(), contrary to other PMDs, mlx4 generates
for each port something that is not a PCI bus address precisely to avoid
collisions. The name is the concatenation of the associated Verbs device
name and the physical port number, e.g. "mlx4_0 port 0" instead of
"0000:02:42.0".

Referring to a port with this name through the above function should yield
the correct device.

If necessary, the PCI address to Verbs device name conversion can be
performed like the PMD itself by looking at sysfs, e.g:

 $ PCI=0000:02:42.0
 $ ls /sys/bus/pci/devices/$PCI/infiniband/
 mlx4_0

Since DPDK 17.05, there's also a way to select what ports the mlx4 PMD
should register DPDK devices for by providing a "port" parameter (DPDK
devargs). The default is to enable them all, see:

 http://dpdk.org/browse/dpdk/commit/?id=001a520e419fdbab8d83f572c2a4414c7bc8ed07

This can help for single port use cases where several ports are registered
by the PMD and the wrong one is used by the application.

-- 
Adrien Mazarguil
6WIND

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[Adrien Mazarguil]

On Thu, Sep 21, 2017 at 11:17:59AM +0200, Adrien Mazarguil wrote:
> On Wed, Sep 20, 2017 at 06:33:43PM +0100, Kevin Traynor wrote:
> > On 09/08/2017 10:56 AM, Loftus, Ciara wrote:
> > >> Hi,
> > >>
> > >> I have compiled and built ovs-dpdk using DPDK v17.08 and OVS v2.8.0. The
> > >> NIC that I am using is Mellanox ConnectX-3 Pro, which is a dual port 10G
> > >> NIC. The problem with this NIC is that it provides only one PCI address for
> > >> both the 10G ports.
> > >>
> > >> So when I am trying to add the two DPDK ports to my br0 bridge
> > >>
> > >> # ovs-vsctl --no-wait add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> > >> options:dpdk-devargs=0002:01:00.0
> > >>
> > >> # ovs-vsctl --no-wait add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk
> > >> options:dpdk-devargs=0002:01:00.0
> > >>
> > >> The port dpdk1 is added successfully and able to transfer data, but adding
> > >> dpdk0 to br0 fails:
> > >>
> > >> 2017-09-06T14:19:20Z|00045|netdev_dpdk|INFO|Port 0: e4:1d:2d:4f:78:60
> > >> 2017-09-06T14:19:20Z|00046|bridge|INFO|bridge br0: added interface dpdk1
> > >> on
> > >> port 1
> > >> 2017-09-06T14:19:20Z|00047|bridge|INFO|bridge br0: added interface br0
> > >> on
> > >> port 65534
> > >> 2017-09-06T14:19:20Z|00048|dpif_netlink|WARN|Generic Netlink family
> > >> 'ovs_datapath' does not exist. The Open vSwitch kernel module is probably
> > >> not loaded.
> > >> 2017-09-06T14:19:20Z|00049|netdev_dpdk|WARN|'dpdk0' is trying to use
> > >> device
> > >> '0002:01:00.0' which is already in use by 'dpdk1'
> > >> 2017-09-06T14:19:20Z|00050|netdev|WARN|dpdk0: could not set
> > >> configuration
> > >> (Address already in use)
> > >> 2017-09-06T14:19:20Z|00051|bridge|INFO|bridge br0: using datapath ID
> > >> 0000e41d2d4f7860
> > >>
> > >>
> > >> With OVS v2.6.1 I never had this problem as dpdk-devargs was not
> > >> mandatory
> > >> and just specifying port name was enough to add that port to bridge.
> > >>
> > >> Is there a way to add port both ports to bridge ?
> > > 
> > > It seems the DPDK function rte_eth_dev_get_port_by_name() will always return the port ID of the first port on your NIC, when you specify the single PCI address and that's where the problem is. There doesn't seem to be a way currently to indicate to the calling application that in fact two (or more) port IDs are associated with the one PCI address.
> > > 
> > > I am cc-ing DPDK users mailing list for hopefully some input. Are there any plans for the rte_eth_dev_get_port_by_name function to be compatible with NICs with multiple ports under the same PCI address?
> > > 
> > 
> > Hi Adrien/Nelio,
> > 
> > Is this something you can answer? We're wondering how to handle this in
> > OVS and whether a temporary or long term solution is needed.
> > 
> > The original thread started here:
> > https://mail.openvswitch.org/pipermail/ovs-dev/2017-September/338418.html
> 
> Hi,
> 
> I'm late to this thread and not too familiar with openvswitch's internals,
> however after reading this thread here are some thoughts.
> 
> About rte_eth_dev_get_port_by_name(), contrary to other PMDs, mlx4 generates
> for each port something that is not a PCI bus address precisely to avoid
> collisions. The name is the concatenation of the associated Verbs device
> name and the physical port number, e.g. "mlx4_0 port 0" instead of
> "0000:02:42.0".
> 
> Referring to a port with this name through the above function should yield
> the correct device.

Well, I replied before checking the current code.

Interpret the above "should" as "would have been nice if it". The device
goes by several names, and since commit [1] the name set by the PMD through
rte_eth_dev_allocate() (struct rte_eth_dev.data->name) is not the same as
the one used by rte_eth_dev_allocated() and rte_eth_dev_get_port_by_name()
(struct rte_eth_dev.device->name).

Without further intervention, the above suggestion won't work in DPDK 17.11.

[1] http://dpdk.org/browse/dpdk/commit/?id=a1e7c17555e8f77d520ba5f06ed26c00e77a2bd1

> If necessary, the PCI address to Verbs device name conversion can be
> performed like the PMD itself by looking at sysfs, e.g:
> 
>  $ PCI=0000:02:42.0
>  $ ls /sys/bus/pci/devices/$PCI/infiniband/
>  mlx4_0
> 
> Since DPDK 17.05, there's also a way to select what ports the mlx4 PMD
> should register DPDK devices for by providing a "port" parameter (DPDK
> devargs). The default is to enable them all, see:
> 
>  http://dpdk.org/browse/dpdk/commit/?id=001a520e419fdbab8d83f572c2a4414c7bc8ed07
> 
> This can help for single port use cases where several ports are registered
> by the PMD and the wrong one is used by the application.

This workaround still stands though.

-- 
Adrien Mazarguil
6WIND

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[devendra rawat]

On Thu, Sep 21, 2017 at 1:58 PM, Loftus, Ciara <ciara.loftus@intel.com>
wrote:

> > 21/09/2017 10:04, Loftus, Ciara:
> > > > 20/09/2017 19:33, Kevin Traynor:
> > > > > On 09/08/2017 10:56 AM, Loftus, Ciara wrote:
> > > > > > It seems the DPDK function rte_eth_dev_get_port_by_name() will
> > > > > > always return the port ID of the first port on your NIC,
> > > > > > when you specify the single PCI address and that's where the
> > > > > > problem is. There doesn't seem to be a way currently to
> > > > > > indicate to the calling application that in fact two
> > > > > > (or more) port IDs are associated with the one PCI address.
> > > >
> > > > We have two ports (with the same PCI address) so we should have
> > > > two different names.
> > > > Where the names passed to rte_eth_dev_get_port_by_name() come
> > from?
> > > > It is the user parameter from options:dpdk-devargs=0002:01:00.0,
> right?
> > >
> > > Yes, we're using the PCI address specified by the user in dpdk-devargs.
> > >
> > > > > > I am cc-ing DPDK users mailing list for hopefully some input.
> > > > > > Are there any plans for the rte_eth_dev_get_port_by_name
> > function
> > > > > > to be compatible with NICs with multiple ports under the same PCI
> > address?
> > > >
> > > > We cannot return two different ports for the same name.
> > > > There are two issues here:
> > > >   - the input should not be the PCI address
> > > >   - the ethdev function should look at ethdev name, not rte_device
> > > > one
> > >
> > > This would require the user having to "guess" the DPDK ethdev name
> > > which is something we'd like to avoid.
> >
> > Yes, but you can provide a way to list the ports with their names
> > and characteristics.
>
> Ok, I see. Maybe something like this could be considered:
>
> port A = dpdk-devargs=xx:yy:zz 0
> port B = dpdk-devargs=xx:yy:zz 1
>
> If we detect a value after the PCI address we iterate through the
> rte_eth_dev_info (http://dpdk.org/doc/api/structrte__eth__dev__info.html)
> for all valid port IDs and assign port A to the first ethdev encountered
> with the provided PCI address, and port B to the second, etc.
>
> If we don't detect a value, then we operate as normal. Thoughts?
>
>
Hi Everyone,

Anything finalized for sorting out this issue, do you need any more
information regarding this issue ?

Thanks,
Devendra


> >
> > > We had the same problem using DPDK port IDs and decided not to use
> > > them anymore, and use the PCI instead as it took the guesswork out.
> > > Ethdev names and port IDs can change between tests, unlike the PCI
> > > address which tends to remain constant for a device.
> >
> > We can add a requirement on ethdev names and make sure they remain
> > constant for a given port.
> >
> > > > The idea is that we have only one rte_device object and we
> instantiate
> > > > two rte_eth_dev ports.
> > > > An ethdev port can be identified with its id (a number) or its unique
> > name.
> > > > Unfortunately, the user cannot guess the port id or the name set by
> the
> > > > PMD.
> > >
> > > Exactly. Thanks for clarifying what's going on under the hood.
> > >
> > > Ciara
> > >
> > > >
> > > > > Hi Adrien/Nelio,
> > > > >
> > > > > Is this something you can answer? We're wondering how to handle
> this
> > in
> > > > > OVS and whether a temporary or long term solution is needed.
> > > >
> > > > I suggest to rely on ethdev name.
> > > > You will need to show to the user the mapping between the bus
> > information
> > > > (PCI id here) and the device names.
> > > >
> > > > Another alternative is to add a new function returning all ethdev
> ports
> > > > associated to a given rte_device resource.
> > > > So you would get two ports and you could pick one on the first "add-
> > port",
> > > > and the other one for the second "add-port" command.
> > > > It means the user would be forced to add them in the right order if
> he
> > > > wants a reproducible result.
>
>

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[Loftus, Ciara]

> 
> On Thu, Sep 21, 2017 at 1:58 PM, Loftus, Ciara <ciara.loftus@intel.com>
> wrote:
> > 21/09/2017 10:04, Loftus, Ciara:
> > > > 20/09/2017 19:33, Kevin Traynor:
> > > > > On 09/08/2017 10:56 AM, Loftus, Ciara wrote:
> > > > > > It seems the DPDK function rte_eth_dev_get_port_by_name() will
> > > > > > always return the port ID of the first port on your NIC,
> > > > > > when you specify the single PCI address and that's where the
> > > > > > problem is. There doesn't seem to be a way currently to
> > > > > > indicate to the calling application that in fact two
> > > > > > (or more) port IDs are associated with the one PCI address.
> > > >
> > > > We have two ports (with the same PCI address) so we should have
> > > > two different names.
> > > > Where the names passed to rte_eth_dev_get_port_by_name() come
> > from?
> > > > It is the user parameter from options:dpdk-devargs=0002:01:00.0,
> right?
> > >
> > > Yes, we're using the PCI address specified by the user in dpdk-devargs.
> > >
> > > > > > I am cc-ing DPDK users mailing list for hopefully some input.
> > > > > > Are there any plans for the rte_eth_dev_get_port_by_name
> > function
> > > > > > to be compatible with NICs with multiple ports under the same PCI
> > address?
> > > >
> > > > We cannot return two different ports for the same name.
> > > > There are two issues here:
> > > >   - the input should not be the PCI address
> > > >   - the ethdev function should look at ethdev name, not rte_device
> > > > one
> > >
> > > This would require the user having to "guess" the DPDK ethdev name
> > > which is something we'd like to avoid.
> >
> > Yes, but you can provide a way to list the ports with their names
> > and characteristics.
> Ok, I see. Maybe something like this could be considered:
> 
> port A = dpdk-devargs=xx:yy:zz 0
> port B = dpdk-devargs=xx:yy:zz 1
> 
> If we detect a value after the PCI address we iterate through the
> rte_eth_dev_info
> (http://dpdk.org/doc/api/structrte__eth__dev__info.html) for all valid port
> IDs and assign port A to the first ethdev encountered with the provided PCI
> address, and port B to the second, etc.
> 
> If we don't detect a value, then we operate as normal. Thoughts?
> 
> Hi Everyone,
> 
> Anything finalized for sorting out this issue, do you need any more
> information regarding this issue ?

Hi,

I put together a very rough RFC that aims to work-around the issue:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-October/339496.html

It hasn't been tested as I don't have access to the type of card that has one PCI for multiple ports.
If anybody does have access to such a device, I welcome you to try the patch although I'm not hopeful it will succeed first -pass.
Looking for feedback on implementation, interface, etc.

Thanks,
Ciara

> 
> Thanks,
> Devendra
> 
> 
> >
> > > We had the same problem using DPDK port IDs and decided not to use
> > > them anymore, and use the PCI instead as it took the guesswork out.
> > > Ethdev names and port IDs can change between tests, unlike the PCI
> > > address which tends to remain constant for a device.
> >
> > We can add a requirement on ethdev names and make sure they remain
> > constant for a given port.
> >
> > > > The idea is that we have only one rte_device object and we instantiate
> > > > two rte_eth_dev ports.
> > > > An ethdev port can be identified with its id (a number) or its unique
> > name.
> > > > Unfortunately, the user cannot guess the port id or the name set by the
> > > > PMD.
> > >
> > > Exactly. Thanks for clarifying what's going on under the hood.
> > >
> > > Ciara
> > >
> > > >
> > > > > Hi Adrien/Nelio,
> > > > >
> > > > > Is this something you can answer? We're wondering how to handle
> this
> > in
> > > > > OVS and whether a temporary or long term solution is needed.
> > > >
> > > > I suggest to rely on ethdev name.
> > > > You will need to show to the user the mapping between the bus
> > information
> > > > (PCI id here) and the device names.
> > > >
> > > > Another alternative is to add a new function returning all ethdev ports
> > > > associated to a given rte_device resource.
> > > > So you would get two ports and you could pick one on the first "add-
> > port",
> > > > and the other one for the second "add-port" command.
> > > > It means the user would be forced to add them in the right order if he
> > > > wants a reproducible result.

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[devendra rawat]

Hi Ciara,

On Fri, Oct 6, 2017 at 7:30 PM, Loftus, Ciara <ciara.loftus@intel.com>
wrote:

> >
> > On Thu, Sep 21, 2017 at 1:58 PM, Loftus, Ciara <ciara.loftus@intel.com>
> > wrote:
> > > 21/09/2017 10:04, Loftus, Ciara:
> > > > > 20/09/2017 19:33, Kevin Traynor:
> > > > > > On 09/08/2017 10:56 AM, Loftus, Ciara wrote:
> > > > > > > It seems the DPDK function rte_eth_dev_get_port_by_name() will
> > > > > > > always return the port ID of the first port on your NIC,
> > > > > > > when you specify the single PCI address and that's where the
> > > > > > > problem is. There doesn't seem to be a way currently to
> > > > > > > indicate to the calling application that in fact two
> > > > > > > (or more) port IDs are associated with the one PCI address.
> > > > >
> > > > > We have two ports (with the same PCI address) so we should have
> > > > > two different names.
> > > > > Where the names passed to rte_eth_dev_get_port_by_name() come
> > > from?
> > > > > It is the user parameter from options:dpdk-devargs=0002:01:00.0,
> > right?
> > > >
> > > > Yes, we're using the PCI address specified by the user in
> dpdk-devargs.
> > > >
> > > > > > > I am cc-ing DPDK users mailing list for hopefully some input.
> > > > > > > Are there any plans for the rte_eth_dev_get_port_by_name
> > > function
> > > > > > > to be compatible with NICs with multiple ports under the same
> PCI
> > > address?
> > > > >
> > > > > We cannot return two different ports for the same name.
> > > > > There are two issues here:
> > > > >   - the input should not be the PCI address
> > > > >   - the ethdev function should look at ethdev name, not rte_device
> > > > > one
> > > >
> > > > This would require the user having to "guess" the DPDK ethdev name
> > > > which is something we'd like to avoid.
> > >
> > > Yes, but you can provide a way to list the ports with their names
> > > and characteristics.
> > Ok, I see. Maybe something like this could be considered:
> >
> > port A = dpdk-devargs=xx:yy:zz 0
> > port B = dpdk-devargs=xx:yy:zz 1
> >
> > If we detect a value after the PCI address we iterate through the
> > rte_eth_dev_info
> > (http://dpdk.org/doc/api/structrte__eth__dev__info.html) for all valid
> port
> > IDs and assign port A to the first ethdev encountered with the provided
> PCI
> > address, and port B to the second, etc.
> >
> > If we don't detect a value, then we operate as normal. Thoughts?
> >
> > Hi Everyone,
> >
> > Anything finalized for sorting out this issue, do you need any more
> > information regarding this issue ?
>
> Hi,
>
> I put together a very rough RFC that aims to work-around the issue:
> https://mail.openvswitch.org/pipermail/ovs-dev/2017-October/339496.html
>
> It hasn't been tested as I don't have access to the type of card that has
> one PCI for multiple ports.
> If anybody does have access to such a device, I welcome you to try the
> patch although I'm not hopeful it will succeed first -pass.
> Looking for feedback on implementation, interface, etc.
>
>  I applied the patch on top of OVS v2.8.1 and used DPDK v17.08. I used
Mellanox ConnectX-3 pro NIC for testing, this NIC provides two 10G ports
that share a single PCI address.

The patch is working fine, I was able to add both the 10G ports to OVS
bridge by specifying the port no. (0 or 1) in the dpdk-devargs.

# ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
options:dpdk-devargs=0002:01:00.0,0
# ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
options:dpdk-devargs=0002:01:00.0,1

the port no. (0 and 1) are bound to actual physical port, i.e if I keep on
adding and deleting port no. 0 to bridge br0 multiple times,
every time I add the port back to br0, the same physical port is added to
bridge br0.

Thanks,
Devendra


> >
> > Thanks,
> > Devendra
> >
> >
> > >
> > > > We had the same problem using DPDK port IDs and decided not to use
> > > > them anymore, and use the PCI instead as it took the guesswork out.
> > > > Ethdev names and port IDs can change between tests, unlike the PCI
> > > > address which tends to remain constant for a device.
> > >
> > > We can add a requirement on ethdev names and make sure they remain
> > > constant for a given port.
> > >
> > > > > The idea is that we have only one rte_device object and we
> instantiate
> > > > > two rte_eth_dev ports.
> > > > > An ethdev port can be identified with its id (a number) or its
> unique
> > > name.
> > > > > Unfortunately, the user cannot guess the port id or the name set
> by the
> > > > > PMD.
> > > >
> > > > Exactly. Thanks for clarifying what's going on under the hood.
> > > >
> > > > Ciara
> > > >
> > > > >
> > > > > > Hi Adrien/Nelio,
> > > > > >
> > > > > > Is this something you can answer? We're wondering how to handle
> > this
> > > in
> > > > > > OVS and whether a temporary or long term solution is needed.
> > > > >
> > > > > I suggest to rely on ethdev name.
> > > > > You will need to show to the user the mapping between the bus
> > > information
> > > > > (PCI id here) and the device names.
> > > > >
> > > > > Another alternative is to add a new function returning all ethdev
> ports
> > > > > associated to a given rte_device resource.
> > > > > So you would get two ports and you could pick one on the first
> "add-
> > > port",
> > > > > and the other one for the second "add-port" command.
> > > > > It means the user would be forced to add them in the right order
> if he
> > > > > wants a reproducible result.
>
>

Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge

[Yuanhan Liu]

On Tue, Oct 10, 2017 at 06:26:51PM +0530, devendra rawat wrote:
>     > Hi Everyone,
>     >
>     > Anything finalized for sorting out this issue, do you need any more
>     > information regarding this issue ?
> 
>     Hi,
> 
>     I put together a very rough RFC that aims to work-around the issue:
>     https://mail.openvswitch.org/pipermail/ovs-dev/2017-October/339496.html
> 
>     It hasn't been tested as I don't have access to the type of card that has
>     one PCI for multiple ports.
>     If anybody does have access to such a device, I welcome you to try the
>     patch although I'm not hopeful it will succeed first -pass.
>     Looking for feedback on implementation, interface, etc.
> 
> 
>  I applied the patch on top of OVS v2.8.1 and used DPDK v17.08. I used Mellanox
> ConnectX-3 pro NIC for testing, this NIC provides two 10G ports
> that share a single PCI address.
> 
> The patch is working fine, I was able to add both the 10G ports to OVS bridge
> by specifying the port no. (0 or 1) in the dpdk-devargs.
> 
> # ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> options:dpdk-devargs=0002:01:00.0,0
> # ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> options:dpdk-devargs=0002:01:00.0,1

I think the patch should work. But there something I want to point out:

- you were assuming the ports with the same PCI addresses are continuous
  in the port number. I don't think it's a good assumption, though it's
  very likely it's always the case in reality.

- I also think the index can not tell which port it is from the ports
  having the same PCI address.

Having said that, I think using the mac address to referencing a DPDK port
might be a better idea. Something like:

    options:dpdk-devargs=$mac_addr

Then from the ovs-dpdk point of you, it can detect whether it's a mac
or an PCI address, and use that to get the right port.

Or, we could have it both, if that's really needed:

    dpdk-devargs=$pci_addr,$mac_addr

In such way, the original issue would also be solved. Moreover, people
know exactly what the underlie port he want to deal with.

Comments/thought?

	--yliu

> 
> the port no. (0 and 1) are bound to actual physical port, i.e if I keep on
> adding and deleting port no. 0 to bridge br0 multiple times,
> every time I add the port back to br0, the same physical port is added to
> bridge br0.
>