From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f193.google.com (mail-wr0-f193.google.com [209.85.128.193]) by dpdk.org (Postfix) with ESMTP id 7D7081B36D for ; Thu, 18 Jan 2018 18:58:10 +0100 (CET) Received: by mail-wr0-f193.google.com with SMTP id 100so23471515wrb.7 for ; Thu, 18 Jan 2018 09:58:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=qFYbCVjOqQTbJJ6uTxn7ECeN3weSs2wk/I6+GxJC82o=; b=0es74oxKo7GrjOncxz/w34UIXdiFTT0i7G2M70toy2R+i/7I7aLvoDcsO3nsXzvbwJ tHD9IuDyY9sRi5FnB9Um6hrIDObDNZuwUL2PI9PRqpeRDNa0FruyeS35wXUMlcS5murN JoHHTqr2DDWEUJtagLjQkDQ+49cmEuDGkeDwFskZ2r2uz3LB5CicddodzN1Z/pxWdWy/ 1xHOTehdrSbkGSlEVu1bDAGjMAKWUw5IuygRXB47Wqkbos6Z60TEeJDdfVhxCBu65gXq Tewkmrs9f3nwddDEllK3I3ej7BFK+/GfBTJS8T+IE8F0BgT/tjCrXh95dGlK9Bdw4Vh8 C6sQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=qFYbCVjOqQTbJJ6uTxn7ECeN3weSs2wk/I6+GxJC82o=; b=LlVtK9fbSnvgoBPF87tQGnaBVqEF9NCe0inA1KlEnmoK3QqIUrzcfuHnWjublYsKNg KZhzeXsKYjZOb6EHJ8uh83IzSS9uvgw6Snk6qtVJNWMggn6eKbCG+OLhkaIvrbFAXvpy giCOYcrr7jYHunxn3wTSLYr0GLnz5izymAQMugAZPinKL4MOBwfHz3eZD617ZJncFsMM 5Po+zKaIVxakfSvmTcc0mzChHBcIx0pjzxTJxOTvpLLSV+KQcCvOs5dFED9LemS27oWr 0y3Y6o1ba+fKQI/pavgxpeQO4mkdhb/zcn4BNdcsvOFnwtpa8MVcJ7bI4Rtn6ttfvKri DB4Q== X-Gm-Message-State: AKwxytdnXQ01qZz1nbQxngPNHickjcbxtB2ZbfpgvGk647mWo8HsRyhA kwJP/+dJRMwnSuyK9QtmB7xeVw== X-Google-Smtp-Source: ACJfBos/+jn41uLV09QTjfG0FedpTBTrNOisCY7JpS/Y5pspQzvDXPTdQN6k+2+MmoUWv6DXkIQHeA== X-Received: by 10.223.161.150 with SMTP id u22mr7536242wru.251.1516298290111; Thu, 18 Jan 2018 09:58:10 -0800 (PST) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id m69sm1997134wma.3.2018.01.18.09.58.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Jan 2018 09:58:09 -0800 (PST) Date: Thu, 18 Jan 2018 18:57:57 +0100 From: Adrien Mazarguil To: Ferruh Yigit Cc: Matan Azrad , Gaetan Rivet , Thomas Monjalon , dev@dpdk.org Message-ID: <20180118175757.GT4256@6wind.com> References: <1516220357-13013-1-git-send-email-matan@mellanox.com> <1516274834-19755-1-git-send-email-matan@mellanox.com> <1516274834-19755-2-git-send-email-matan@mellanox.com> <2ffac5e9-6be5-0c82-18c4-8b72710630ae@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2ffac5e9-6be5-0c82-18c4-8b72710630ae@intel.com> Subject: Re: [dpdk-dev] [PATCH v6 1/6] ethdev: add devop to check removal status X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jan 2018 17:58:10 -0000 On Thu, Jan 18, 2018 at 05:18:22PM +0000, Ferruh Yigit wrote: > On 1/18/2018 11:27 AM, Matan Azrad wrote: > > There is time between the physical removal of the device until PMDs get > > a RMV interrupt. At this time DPDK PMDs and applications still don't > > know about the removal. > > > > Current removal detection is achieved only by registration to device RMV > > event and the notification comes asynchronously. So, there is no option > > to detect a device removal synchronously. > > Applications and other DPDK entities may want to check a device removal > > synchronously and to take an immediate decision accordingly. > > So we will have two methods to detect device removal, one is asynchronous as you > mentioned. > Device removal will cause an interrupt which trigger to run user callback. > > New method is synchronous, but still triggered from application. I mean > application should do a rte_eth_dev_is_removed() to learn about status, what is > the use case here, polling continuously? Won't this also cause some latency > unless you dedicate a core just polling device status? They are complementary. The use case is when devices get suddenly physically pulled out of their chassis (you need to picture a raging sysadmin for that), or logically in the case of a hypervisor removing a SR-IOV device from a VM, this happens without prior notice. It takes time for the PCI unplug notification to travel from the kernel to DPDK, up to several seconds, during which the DPDK application may execute control path operations on it. These may fail due to the now non-existent device (e.g. no ACK will be returned by the device after adding a new MAC), and these failures may be misinterpreted (e.g. permission denied, invalid argument and so on). To address this problem, PMDs that support physical hotplug must have all their devops internally check for device removal before returning any other error, in order to possibly convert the original error code to EIO. Now patching each and every devop in each PMD with basically the same code being counterproductive, this series puts this check at a higher level, inside rte_ethdev. Since this results in a new devop, it can be exposed to applications for free, as these may find a use for it as well. > > Add new dev op called is_removed to allow DPDK entities to check an > > Ethernet device removal status immediately. > > > > Signed-off-by: Matan Azrad > > Acked-by: Thomas Monjalon > > --- > > lib/librte_ether/rte_ethdev.c | 28 +++++++++++++++++++++++++--- > > lib/librte_ether/rte_ethdev.h | 20 ++++++++++++++++++++ > > lib/librte_ether/rte_ethdev_version.map | 1 + > > 3 files changed, 46 insertions(+), 3 deletions(-) > > > > diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c > > index b349599..c93cec1 100644 > > --- a/lib/librte_ether/rte_ethdev.c > > +++ b/lib/librte_ether/rte_ethdev.c > > @@ -114,7 +114,8 @@ enum { > > rte_eth_find_next(uint16_t port_id) > > { > > while (port_id < RTE_MAX_ETHPORTS && > > - rte_eth_devices[port_id].state != RTE_ETH_DEV_ATTACHED) > > + rte_eth_devices[port_id].state != RTE_ETH_DEV_ATTACHED && > > + rte_eth_devices[port_id].state != RTE_ETH_DEV_REMOVED) > > If device is removed, why we are not allowed to re-use port_id assigned to it? > Overall I am not clear with RTE_ETH_DEV_REMOVED state, why we are not directly > setting RTE_ETH_DEV_UNUSED? > > And state RTE_ETH_DEV_REMOVED set in ethdev layer, and ethdev layer won't let > reusing it, so what changes the state of dev? Will it stay as it is during > lifetime of the application? While it switched to the REMOVED state, the underlying PMD still holds the entry at this point; data is still allocated and so on. It will switch to UNUSED after the PMD instance is fully de-initialized. In the meantime the entry still needs to be skipped. > > port_id++; > > > > if (port_id >= RTE_MAX_ETHPORTS) > > @@ -262,8 +263,7 @@ struct rte_eth_dev * > > rte_eth_dev_is_valid_port(uint16_t port_id) > > { > > if (port_id >= RTE_MAX_ETHPORTS || > > - (rte_eth_devices[port_id].state != RTE_ETH_DEV_ATTACHED && > > - rte_eth_devices[port_id].state != RTE_ETH_DEV_DEFERRED)) > > + (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED)) > > return 0; > > else > > return 1; > > @@ -1094,6 +1094,28 @@ struct rte_eth_dev * > > } > > > > int > > +rte_eth_dev_is_removed(uint16_t port_id) > > +{ > > + struct rte_eth_dev *dev; > > + int ret; > > + > > + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0); > > + > > + dev = &rte_eth_devices[port_id]; > > + > > + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->is_removed, 0); > > + > > + if (dev->state == RTE_ETH_DEV_REMOVED) > > + return 1; > > Isn't this conflict with below API documentation: > > " > * @return > * - 0 when the Ethernet device is removed, otherwise 1. > " Documentation is indeed wrong here. Matan? > > > + > > + ret = dev->dev_ops->is_removed(dev); > > + if (ret != 0) > > + dev->state = RTE_ETH_DEV_REMOVED; > > It isn't clear what "dev_ops->is_removed(dev)" should return, and this causing > incompatible usages in PMDs by time. > Please add some documentation about expected return values for dev_ops. It should be clarified as a boolean value (yes = nonzero, no = zero), like most is*() functions (isalpha(), isblank() and so on). > And this not real remove, PMD signals us and we stop using that device, but > device can be there, right? "Removal" in the sense of "device removal" not "PMD removal" which is usually described as "unbinding". This was chosen based on the similarly-named "removal" (RMV) event for consistency. > If there is a real removal, can be possible to use eal hotplug? Possibly, although I think it doesn't remove the case for this devop, right? -- Adrien Mazarguil 6WIND