From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f179.google.com (mail-wr0-f179.google.com [209.85.128.179]) by dpdk.org (Postfix) with ESMTP id 8525D1B027 for ; Thu, 21 Sep 2017 11:45:39 +0200 (CEST) Received: by mail-wr0-f179.google.com with SMTP id 108so4116364wra.5 for ; Thu, 21 Sep 2017 02:45:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=l1l64SMfOxz4agEipfJtvm1kjPEe8Vck0KL5gZtRhK8=; b=xFipfjKIxoXV50dZYASjawUOTxtsGTqyUff7p6MPcCap7AW+TPmvCPTFCwhbAYKx+R +R5SWK+qIsDM/XNtVP1TD/eVdzSTgfPWn1ae39ssql94vEwLnBU5UkyGavZTZ1SXFI+V rvqjRgh9yvSP7cK3P2cKYBZTCksZoGGnBHS2XsdH93qkgAney8lDIY5q/aApCi1UqWV9 BVnc9uHMzMGuo/JlbZS3f8mZVB11oDLMWT8Pc/x1LZi9pyOrQa0X7QnD4ymGXWJUo8HP h07mVQC/ndvRYJsZHMt+NVZYJSmaUykiIpEsEWxcDwd1vm3B8D36bW7wNw+HLJ3r5xzV 2TXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=l1l64SMfOxz4agEipfJtvm1kjPEe8Vck0KL5gZtRhK8=; b=lP/EiyaFnA9VHN3K0NkfVFBYj9syLXMj5mfIFSc+h46CSxREUAuR8yykshXBsqGs0P 1hR/t8hs561vjY2bjUrG0A44l6kp4ltKTrTgyepHzXGJjv8Co0Cy/ojbLswUbmoQ7wAR o2Rh8swBBoTk/UQw54fXAOES1/ct9PNycdHaTiywnFWGeoklVD5sY1Q2Ha3iBfCKw1fp Sprz+LD4R4BwaX75lzzagHbekK3+6Yhqgdc2roEqfZ8NmidnpBJYHJUNYw88jsumvRtB aqr19gn2Wa5apViEGL0q1FRz72zr1o/CBW8ZE13DegzArUbAetbJCoHKk8+4ssH5NfFY K14g== X-Gm-Message-State: AHPjjUjRFyo4MxsyD3CK2wZMoakaA9/bjoR0Uo/23Rty0B+XrK3pZLXi Q401ea4TmF2PFtEJXn4z3c1b3Q== X-Google-Smtp-Source: AOwi7QCg3Fol7FwHwLQMLuGzlfux7H4ySZBWyYVhCDMuBmbRbexZXtXR0ifY3Ds2kMBVODD1vhwUWg== X-Received: by 10.223.178.193 with SMTP id g59mr1468845wrd.0.1505987139055; Thu, 21 Sep 2017 02:45:39 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id m128sm3276086wmf.0.2017.09.21.02.45.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Sep 2017 02:45:37 -0700 (PDT) Date: Thu, 21 Sep 2017 11:45:28 +0200 From: Adrien Mazarguil To: Kevin Traynor Cc: "Loftus, Ciara" , devendra rawat , "ovs-dev@openvswitch.org" , nelio.laranjeiro@6wind.com, "users@dpdk.org" , Yuanhan Liu , Thomas Monjalon Message-ID: <20170921094528.GB11375@6wind.com> References: <74F120C019F4A64C9B78E802F6AD4CC278E04762@IRSMSX106.ger.corp.intel.com> <20170921091759.GA11375@6wind.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170921091759.GA11375@6wind.com> Subject: Re: [dpdk-users] [ovs-dev] adding dpdk ports sharing same pci address to ovs-dpdk bridge X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Sep 2017 09:45:39 -0000 On Thu, Sep 21, 2017 at 11:17:59AM +0200, Adrien Mazarguil wrote: > On Wed, Sep 20, 2017 at 06:33:43PM +0100, Kevin Traynor wrote: > > On 09/08/2017 10:56 AM, Loftus, Ciara wrote: > > >> Hi, > > >> > > >> I have compiled and built ovs-dpdk using DPDK v17.08 and OVS v2.8.0. The > > >> NIC that I am using is Mellanox ConnectX-3 Pro, which is a dual port 10G > > >> NIC. The problem with this NIC is that it provides only one PCI address for > > >> both the 10G ports. > > >> > > >> So when I am trying to add the two DPDK ports to my br0 bridge > > >> > > >> # ovs-vsctl --no-wait add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk > > >> options:dpdk-devargs=0002:01:00.0 > > >> > > >> # ovs-vsctl --no-wait add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk > > >> options:dpdk-devargs=0002:01:00.0 > > >> > > >> The port dpdk1 is added successfully and able to transfer data, but adding > > >> dpdk0 to br0 fails: > > >> > > >> 2017-09-06T14:19:20Z|00045|netdev_dpdk|INFO|Port 0: e4:1d:2d:4f:78:60 > > >> 2017-09-06T14:19:20Z|00046|bridge|INFO|bridge br0: added interface dpdk1 > > >> on > > >> port 1 > > >> 2017-09-06T14:19:20Z|00047|bridge|INFO|bridge br0: added interface br0 > > >> on > > >> port 65534 > > >> 2017-09-06T14:19:20Z|00048|dpif_netlink|WARN|Generic Netlink family > > >> 'ovs_datapath' does not exist. The Open vSwitch kernel module is probably > > >> not loaded. > > >> 2017-09-06T14:19:20Z|00049|netdev_dpdk|WARN|'dpdk0' is trying to use > > >> device > > >> '0002:01:00.0' which is already in use by 'dpdk1' > > >> 2017-09-06T14:19:20Z|00050|netdev|WARN|dpdk0: could not set > > >> configuration > > >> (Address already in use) > > >> 2017-09-06T14:19:20Z|00051|bridge|INFO|bridge br0: using datapath ID > > >> 0000e41d2d4f7860 > > >> > > >> > > >> With OVS v2.6.1 I never had this problem as dpdk-devargs was not > > >> mandatory > > >> and just specifying port name was enough to add that port to bridge. > > >> > > >> Is there a way to add port both ports to bridge ? > > > > > > It seems the DPDK function rte_eth_dev_get_port_by_name() will always return the port ID of the first port on your NIC, when you specify the single PCI address and that's where the problem is. There doesn't seem to be a way currently to indicate to the calling application that in fact two (or more) port IDs are associated with the one PCI address. > > > > > > I am cc-ing DPDK users mailing list for hopefully some input. Are there any plans for the rte_eth_dev_get_port_by_name function to be compatible with NICs with multiple ports under the same PCI address? > > > > > > > Hi Adrien/Nelio, > > > > Is this something you can answer? We're wondering how to handle this in > > OVS and whether a temporary or long term solution is needed. > > > > The original thread started here: > > https://mail.openvswitch.org/pipermail/ovs-dev/2017-September/338418.html > > Hi, > > I'm late to this thread and not too familiar with openvswitch's internals, > however after reading this thread here are some thoughts. > > About rte_eth_dev_get_port_by_name(), contrary to other PMDs, mlx4 generates > for each port something that is not a PCI bus address precisely to avoid > collisions. The name is the concatenation of the associated Verbs device > name and the physical port number, e.g. "mlx4_0 port 0" instead of > "0000:02:42.0". > > Referring to a port with this name through the above function should yield > the correct device. Well, I replied before checking the current code. Interpret the above "should" as "would have been nice if it". The device goes by several names, and since commit [1] the name set by the PMD through rte_eth_dev_allocate() (struct rte_eth_dev.data->name) is not the same as the one used by rte_eth_dev_allocated() and rte_eth_dev_get_port_by_name() (struct rte_eth_dev.device->name). Without further intervention, the above suggestion won't work in DPDK 17.11. [1] http://dpdk.org/browse/dpdk/commit/?id=a1e7c17555e8f77d520ba5f06ed26c00e77a2bd1 > If necessary, the PCI address to Verbs device name conversion can be > performed like the PMD itself by looking at sysfs, e.g: > > $ PCI=0000:02:42.0 > $ ls /sys/bus/pci/devices/$PCI/infiniband/ > mlx4_0 > > Since DPDK 17.05, there's also a way to select what ports the mlx4 PMD > should register DPDK devices for by providing a "port" parameter (DPDK > devargs). The default is to enable them all, see: > > http://dpdk.org/browse/dpdk/commit/?id=001a520e419fdbab8d83f572c2a4414c7bc8ed07 > > This can help for single port use cases where several ports are registered > by the PMD and the wrong one is used by the application. This workaround still stands though. -- Adrien Mazarguil 6WIND