From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 17CE82BB9 for ; Fri, 16 Jun 2017 15:49:13 +0200 (CEST) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga105.jf.intel.com with ESMTP; 16 Jun 2017 06:49:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.39,347,1493708400"; d="scan'208";a="1161301925" Received: from fyigit-mobl1.ger.corp.intel.com (HELO [10.237.220.91]) ([10.237.220.91]) by fmsmga001.fm.intel.com with ESMTP; 16 Jun 2017 06:49:12 -0700 To: Adrien Mazarguil Cc: dev@dpdk.org References: <2407596e1e05260170f8b0b80806b28b63e9255d.1497439616.git.adrien.mazarguil@6wind.com> <08cf1e01-f0b9-9540-9cc5-0b7fb79d8799@intel.com> <20170616133955.GD1758@6wind.com> From: Ferruh Yigit Message-ID: Date: Fri, 16 Jun 2017 14:49:11 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <20170616133955.GD1758@6wind.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH 2/7] net/mlx4: fix Rx interrupts with multiple ports X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jun 2017 13:49:14 -0000 On 6/16/2017 2:39 PM, Adrien Mazarguil wrote: > On Fri, Jun 16, 2017 at 02:07:54PM +0100, Ferruh Yigit wrote: >> On 6/14/2017 12:49 PM, Adrien Mazarguil wrote: >>> Several Ethernet device structures are allocated on top of a common PCI >>> device for mlx4 adapters with multiple ports. These inherit a common >>> interrupt handle from their parent PCI device, which prevents Rx interrupts >>> from working properly on all ports as their configuration is overwritten. >>> >>> Use a local interrupt handle to address this issue. >> >> Hi Adrien, >> >> I am not clear why local copy required, and main concern from my point >> of view is if this is a common problem for all PMDs and should be >> addressed in higher level? > > This issue only affects PMDs that handle multiple Ethernet ports through a > single PCI device. Such PMDs (like mlx4) identify themselves as PCI drivers > that manually have to register multiple rte_eth_dev instances through > rte_eth_dev_allocate(), which they then have to initialize. > >> The variable is already per eth_dev, but this patch moves it the private >> data. What overwrites it within eth_dev? > > Calling rte_eth_copy_pci_info() makes the rte_eth_dev structure inherit the > default interrupt handle of the underlying PCI device. By "inherit", I mean > eth_dev->intr_handle points to it, in that sense it's not per eth_dev but > per PCI device. > > mlx4 Rx interrupts are associated with a given Verbs context, and each port > has its own Verbs context, so they cannot be shared, while other PMDs using > other methods for catching interrupts may be perfectly fine with a single > vector associated with the PCI device. It depends on the PMD, for instance > there is no such problem with mlx5 as exactly one PCI device is associated > with a given port. > > This patch merely allocates a specific interrupt handle associated with the > eth_dev itself and makes the eth_dev handle point to that instead of the > default PCI handle. This "local" handle is initialized using the PCI handle > as a template before modifying the pointer. It's completely safe. Thanks for clarification. > >>> Fixes: 9f05a4b81809 ("net/mlx4: support user space Rx interrupt event") >>> >>> Signed-off-by: Adrien Mazarguil >>> Acked-by: Moti Haimovsky >>> --- >>> drivers/net/mlx4/mlx4.c | 9 +++++++++ >>> drivers/net/mlx4/mlx4.h | 1 + >>> 2 files changed, 10 insertions(+) >>> >>> diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c >>> index 178562e..2b4722f 100644 >>> --- a/drivers/net/mlx4/mlx4.c >>> +++ b/drivers/net/mlx4/mlx4.c >>> @@ -6207,6 +6207,15 @@ mlx4_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) >>> >>> eth_dev->device->driver = &mlx4_driver.driver; >>> >>> + /* >>> + * Copy and override interrupt handle to prevent it from >>> + * being shared between all ethdev instances of a given PCI >>> + * device. This is required to properly handle Rx interrupts >>> + * on all ports. >>> + */ >>> + priv->intr_handle_dev = *eth_dev->intr_handle; >>> + eth_dev->intr_handle = &priv->intr_handle_dev; >>> + >>> priv->dev = eth_dev; >>> eth_dev->dev_ops = &mlx4_dev_ops; >>> >>> diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h >>> index c46fc23..b74fbf8 100644 >>> --- a/drivers/net/mlx4/mlx4.h >>> +++ b/drivers/net/mlx4/mlx4.h >>> @@ -345,6 +345,7 @@ struct priv { >>> unsigned int txqs_n; /* TX queues array size. */ >>> struct rxq *(*rxqs)[]; /* RX queues. */ >>> struct txq *(*txqs)[]; /* TX queues. */ >>> + struct rte_intr_handle intr_handle_dev; /* Device interrupt handler. */ >>> struct rte_intr_handle intr_handle; /* Interrupt handler. */ >>> struct rte_flow_drop *flow_drop_queue; /* Flow drop queue. */ >>> LIST_HEAD(mlx4_flows, rte_flow) flows; >>> >> >