From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <Kamil.Rytarowski@cavium.com>
Received: from NAM03-CO1-obe.outbound.protection.outlook.com
 (mail-co1nam03on0052.outbound.protection.outlook.com [104.47.40.52])
 by dpdk.org (Postfix) with ESMTP id 26412677D
 for <dev@dpdk.org>; Fri, 26 Aug 2016 18:55:15 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version;
 bh=qFObSJPLSetJSTg93mc0LIKVO/3VkgEwFil95SuigYo=;
 b=B40scot/XjsHB9aI/Ks464TkBl3F9WLPFZr2GjC3O2wavJZtemep7FO4EnqMWOoL3iP1PGqfQbpAl+JLMt4WqGrUun1P5FXBpPOXjP8jKQJnEE0g1FRN2K6/C1zrIRQtNR8SZvr6zU+42ZTt3AExzWCVqueu7INtQ01sUQdUlpo=
Authentication-Results: spf=none (sender IP is )
 smtp.mailfrom=Kamil.Rytarowski@cavium.com; 
Received: from cavium1.semihalf.local (31.172.191.173) by
 CY4PR07MB3064.namprd07.prod.outlook.com (10.172.116.145) with Microsoft SMTP
 Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id
 15.1.549.15; Fri, 26 Aug 2016 16:55:10 +0000
From: Kamil Rytarowski <krytarowski@caviumnetworks.com>
To: <dev@dpdk.org>
CC: <maciej.czekaj@caviumnetworks.com>, <zyta.szpak@semihalf.com>,
 <slawomir.rosek@semihalf.com>, <rad@semihalf.com>,
 <jerin.jacob@caviumnetworks.com>, Kamil Rytarowski
 <kamil.rytarowski@caviumnetworks.com>
Date: Fri, 26 Aug 2016 18:54:04 +0200
Message-ID: <1472230448-17490-10-git-send-email-krytarowski@caviumnetworks.com>
X-Mailer: git-send-email 1.9.1
In-Reply-To: <1472230448-17490-1-git-send-email-krytarowski@caviumnetworks.com>
References: <1472230448-17490-1-git-send-email-krytarowski@caviumnetworks.com>
MIME-Version: 1.0
Content-Type: text/plain
X-Originating-IP: [31.172.191.173]
X-ClientProxiedBy: HE1PR0201CA0008.eurprd02.prod.outlook.com (10.165.229.18)
 To CY4PR07MB3064.namprd07.prod.outlook.com (10.172.116.145)
X-MS-Office365-Filtering-Correlation-Id: f9aaaf56-7abe-4167-20a1-08d3cdd1bef3
X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064;
 2:U/IOvNmbsudVtmRMYVuuOKEv8g6/5smGuFJb8Werrck1PotiS1idIPBL2RSFC6csuUiqE2y0ysHBTw0XcBA6Uh12jzHJqjBTd3uTq5xq3C09T5tz5bl4Pe3D3jg9gS+8n+HCcmekQnhk0Q3jZIdkBozzNzxm+Y2yayeMSmw8Jc9Tx5W/8m0krZ5hOxPvLCWc;
 3:w6J4punFDFXEpNOwyyt4w3SoAd369rLHL58C86v3PfBYYVex+Ypfxope1kR9Qn2B7ZCX2JkTgrPKgUI6V7wORaaZRW4mQosJe2eWMzzqGp/eeMJul0YgNC2peDRco9BT
X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY4PR07MB3064;
X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064;
 25:Ezcx/1QNORSyr2cJaal6YkiFsK6iTYPd4FeTyRI3NkQ6uC9xnGExda6AiB/DaERmvMX3HH1iqkMLF7OWs6Y3KyFGY7wSmWAWp1nTbP9bEfm6sU+Odicn0npZyOIgHpg9YpKfWsXpQ5VzZ4h3Rl3qa0+k3bAew3QWf06/AxZDeEqJe6SgmnkLEeSE7bjDwgnHUAI0m/8FygwPKFQhATjUtcfykLckwyqIJvcf5H0WWbQxVs/EALenE+MufsolF5w/C43xbNZa6G2uGjYJKLi2JqvZjJelKjCrqL5we9s0nqZFUCP9P6/C/sfqrS6cdyO78WbrBFPPruGXGRUru0v1dvZhVoTVs12l3Um0mMLjf4tdKSe/0EEeCrojjCAO0p8k4f2BU3noo84dR06l4wBUKREO7g3M4DwTmli3siiCmaRtyJx3XEjygdzibIZ+1L8H+IJWpwtGAbmpDhO7WIqeLNNVssoETXfU7TThXuaeUHp3QSEtWEW+A+kJ8HHsMW3fIMzQmSNGXCjQLXvZDojmkG13dVm01erGXhYtI8b5AIr3a1yheH0I8aiPX+5/tKcJLUCMPM6W/i8WDl2kPmqBIIUXr/2kqhgjreDc4GygdJpY3f3bbNOv4o6p11/E2KNM8GKDPfwLFlpCZ9QVh2MiTkmIwPAMqjrHwfdtfTnrTGWoziIKCapAiLKmQRsiYQnRY+RJAE8Nsg3zQo2ye3GVwWEMWB1SdQyqPybtJMcuTEA=
X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064;
 31:EkTH+/6wd5Q7DF227jySAPCSpa40+6y9jlZM1EObIk0detLIVOyd8jHe//RhcIFSepBP92Wp6sxuSNllq89UgQ73GDi9s8gHgaUmYb4ow8veFuYY/3LnYCM46eC/RoXz/qcmH/saTq0eIk5TNJDEEcoVHgF3bug9E+g4EFQTb7y7ghtUx8M+jdFUZ65G+uIh6ledxkB7gbLgqyxvmVrS8dSplKBr4AwiXVEZMr4lp44=;
 20:F00jqyCxcOb9D/8l9EhXpaX2neRWMVlEHVjwIfr/jnAHe03ZO2oQC9c08Mz6nDKn++c2rPIrE5jJOoUQ4YvRCnJUN6iTHkpCKUmHfT73D3JPpCQ77gI3UX1kVkcMbnCXaqIIgziSELDBeKVMzIajXGDcJNwCPu03b2C5t38RN7/cWtbuedOlo99n2GFh6HIqGQiVZ6uezh4bOfC5njvInZrIe5M73XYGtZBoQMOUBbvfCZNdDETvI0r0WrLtXnIWu1t9h9+bE/6Hs9aKvd5bqhXmuz2xtvZEa6VOHR6Mtv2WSJCOCVxVai4vsH+yGTxUoeD1sKw8wThOrKwMQnwIDAkhB8xnysfGwth8Q620bJmVZYPtTbVsFMQlJLcHNg0p7bHXuWPj1qFgnBPUBZEkjIpI0JESLOVDUOJ3NJESlNhdEhCLRxKELXjLHxq/YRIvshcbIX17lwdnpyCae8N0E022cVtI0//wE8KqC4aZg9x/iiLnP+DzFG5X+OEr0Z79PO6DqLmeD+dr7aRE92WCa8nUoFHnQUGf2PAg2KCy0TT75Mbnw1psGorJ4zEzFy2bZoa6FbsK1ilqwR1VTbiPd0yilcsJ0DklziJi2BXgDxU=
X-Microsoft-Antispam-PRVS: <CY4PR07MB306489181CC33C39FD33776AF9EC0@CY4PR07MB3064.namprd07.prod.outlook.com>
X-Exchange-Antispam-Report-Test: UriScan:;
X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0;
 RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001);
 SRVR:CY4PR07MB3064; BCL:0; PCL:0; RULEID:; SRVR:CY4PR07MB3064; 
X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064;
 4:FT7ZH055bP1+enUWpvVdMUxCwYAfbiK34ndd/NV8r0RfTtcHBb7djJzNEdr977vJXrtfHJagHPlR0577VxQhAZGi3u0wuuq02CgjJh3Ep/IXMhRF0EfBWNWv9Ys6o10wK3pWAH5g2/hpRv+BBnLpvZSBSQcIfeI+pQKag0JZ2wQRQpM7Ws5q+pv69By0EWxfy8kiWUSQQ/POCGo4LL4ujYSFcjwbZKrACePAtWBZTs6UW67dYxPMO+cBx8VJfMom+614md7yO0ChHIOYea8SuNJYGru9h7l8b7yXHeT5RzuvL0nquFTF3JK7hQGAygu0r+JNy+9JQTS3GO6k06PyeIOc0KNZ0kOgvaSNdjXKpY+NCL3oVzexWsPcpytBRutWy7Pz9WF4rNqFTtY8J2NVbw==
X-Forefront-PRVS: 00462943DE
X-Forefront-Antispam-Report: SFV:NSPM;
 SFS:(10009020)(4630300001)(6009001)(7916002)(189002)(199003)(81156014)(36756003)(77096005)(189998001)(2906002)(229853001)(2351001)(105586002)(8676002)(101416001)(92566002)(5660300001)(81166006)(6116002)(4001430100002)(2950100001)(7736002)(47776003)(4326007)(66066001)(586003)(50466002)(106356001)(50226002)(5003940100001)(19580405001)(19580395003)(305945005)(50986999)(110136002)(76176999)(7846002)(48376002)(68736007)(42186005)(3846002)(107886002)(33646002)(97736004);
 DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR07MB3064; H:cavium1.semihalf.local; FPR:;
 SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; 
Received-SPF: None (protection.outlook.com: cavium.com does not designate
 permitted sender hosts)
X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR07MB3064;
 23:xF0+RQySLKfB4PAwy55Xb57TyrghNbDN7F24Y25Ay?=
 =?us-ascii?Q?aFliTouoeL2P7sWtiRD8bnJOpzVyvzZlN+fkRjELhWX6wNwG8Lu+7VtbCkWy?=
 =?us-ascii?Q?XRnJOYAomV7cygcje6u7vxFpE5Q/MLjw/R1EDv3Ytuq1JNYi9fEQiEMkeoIw?=
 =?us-ascii?Q?ACCn/UWL4cf1VaIsRLMkqRllVddbYij/NNp1xxbVLcOD+pwDz34JFuuZKf0r?=
 =?us-ascii?Q?AnMSxbgjNXHmRULYqpQIRpiSEskyvcKmnOGjV+BFgw7zyqnkNjzkXV0w+Rwx?=
 =?us-ascii?Q?TAhOu/Cq8UmYSkPJwJxbHULPkdYFQXb5Ccynzf0jmtMGw6DAqY9TX2skYV6v?=
 =?us-ascii?Q?dIEb2yyLzEbmw5pUxd816foOM9o37ZU0FnfHKXC7bGgI9FYq8elb5DK3Z9Ej?=
 =?us-ascii?Q?BXvWIzXRl+xWyyofVQYa/5QEYsEsi+lIVEPCHjOc7Z0lBvTA/9QZrIcGvCoE?=
 =?us-ascii?Q?D2j+AeI03Px633DuaZ07jwfA3pfmnB2NivTCMZrLPQ379B4Xu/VBj7CPxMIm?=
 =?us-ascii?Q?NM6qXxqI/Vlk4GoutWnUoX4T8SS9WU075hfgIdi2vJq+/twnmMR9Ly/A6Guk?=
 =?us-ascii?Q?OOj9OHcBbgEib3WSrt/50SwvjS4KUYsvM6/UqnrKPBt/C/AjEgVZQWfigFKr?=
 =?us-ascii?Q?pkxFdbCcHBQCWybvysp5LMb9oetvRffJ5YIr9ZC7A0dyoHaPcPH2YuHKxRkX?=
 =?us-ascii?Q?yd9HEwMMxFUbeuMfHnbmLRqfrH9kGr36e370EVtjKJfLGRBkq2RZn/TVKCOf?=
 =?us-ascii?Q?F2c0aYt27aXRuHW1pGAp7tDd3UkGTkiu5VHM9iZxMIzJ0zS/AuTBR3+6fzgD?=
 =?us-ascii?Q?NGwl61seWDreJ6H8k2dDc5Rk2Fnf/ykZQ35b8+FsK7+dOq50lDjWZTTYF6vB?=
 =?us-ascii?Q?mPYVkB7GqAiPVz2WODc4esoI7M6noFPDdPfnDtV4ts06/5jAt+mzybfV0mPw?=
 =?us-ascii?Q?qcgJH+BJPKvMQqsd+VMwvAz80YVi1UCShMXeH6zMzdfZ1xd6VN84625FeyIa?=
 =?us-ascii?Q?a9PBgI9TV/lIK+zgHCPHIgU/AHPFjuxPqHWCDWl61de4LA4/xkCBXu0AkXas?=
 =?us-ascii?Q?6P3KAV2Cl113JYJPAZ2CoH3FFzQ+8PWBdgzbbCG1KrmQtBQAtFYjY55H4Swg?=
 =?us-ascii?Q?5kiDoZaH5E=3D?=
X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064;
 6:nIKlRlaFOxx+VqevziXYnnJwZ2hro/L2y8VFzp/dPRL79qEf3evyM/nUls6xyIbBzkGTAm7/nd/TA2XGOZKjEF8RuAWgdK6KEWax4u5+p2F0rKsC1SDu3+BNsnnZ2l7Vz+u8BfOosqtUGqOcHxdnpmS/l6OwyPL32YICmZ/IF9J8arEWe3FxbjMxkqo702Qn+51bvsbhsiSDJhIJDpEzJKDTc3hkDYXRx8+1OeN3O3yYj/lrK7tbkRt8LbuRAbmCCK9m3EoRrT7rktm5KKE6pK+p5PC11taL8BvjS6gFlXY=;
 5:23+x+crM++77b36FDeB/ver3dqgwYLDUCvFGjMiDfNLFi20+Lww7fsbe621ldmoE1W5YR/zcNFxWJbpfoPCrD2LHJ9Zd2HC1qGp3VEvoKqHJt5q80/qYVg7asyk90VKhHmd6PY0H3smKAvowUWp8MQ==;
 24:hY2BDuG0McNIxgp6dpSneutgMu3ryQBP26H4wabmrXTlRPEq4ZVs8VC94f24JyoFst7LDcmM46yMSe6ZqVZQR4uBl+Hn+SR3jXKNnGeEAj0=;
 7:tMUa0M6xyzpjLvHSHZieUUXGWzFsarVg97P/UY06hTJubQl0SFCjuM8cfzQlnc0EYPbebkAxJh8x0DNvatQlFR0JmitOEO20/qzb6Eoa4XERmr3sBAUk/zsbGah8NWOr0JVSmcll9d617FMwyS7VY1QY/QEroB2yCTC+nv4Kk1FrLm4l06QNQr5VHPDak7KMck3iXXkXBYYla8apPnhjnYr6E4I4Vo+Fyu+c7Q2DdWoP6d8QJ4twaZfefR1+c1mp
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-OriginatorOrg: caviumnetworks.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2016 16:55:10.6500 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR07MB3064
Subject: [dpdk-dev] [PATCH 09/13] net/thunderx: add secondary qset support
	in dev stop/close
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 26 Aug 2016 16:55:16 -0000

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 142 +++++++++++++++++++++++++++---------
 1 file changed, 107 insertions(+), 35 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index b4ab505..c9c5b78 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -67,9 +67,13 @@
 
 #include "nicvf_ethdev.h"
 #include "nicvf_rxtx.h"
+#include "nicvf_svf.h"
 #include "nicvf_logs.h"
 
 static void nicvf_dev_stop(struct rte_eth_dev *dev);
+static void nicvf_dev_stop_cleanup(struct rte_eth_dev *dev, bool cleanup);
+static void nicvf_vf_stop(struct rte_eth_dev *dev, struct nicvf *nic,
+			  bool cleanup);
 
 static inline int
 nicvf_atomic_write_link_status(struct rte_eth_dev *dev,
@@ -674,23 +678,29 @@ config_sq_error:
 }
 
 static inline int
-nicvf_stop_tx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_stop_tx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+		       uint16_t qidx)
 {
 	struct nicvf_txq *txq;
 	int ret;
 
-	if (dev->data->tx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STOPPED)
+	assert(qidx < MAX_SND_QUEUES_PER_QS);
+
+	if (dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+		RTE_ETH_QUEUE_STATE_STOPPED)
 		return 0;
 
-	ret = nicvf_qset_sq_reclaim(nicvf_pmd_priv(dev), qidx);
+	ret = nicvf_qset_sq_reclaim(nic, qidx);
 	if (ret)
-		PMD_INIT_LOG(ERR, "Failed to reclaim sq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to reclaim sq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 
-	txq = dev->data->tx_queues[qidx];
+	txq = dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)];
 	nicvf_tx_queue_release_mbufs(txq);
 	nicvf_tx_queue_reset(txq);
 
-	dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+	dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STOPPED;
 	return ret;
 }
 
@@ -1002,30 +1012,34 @@ config_rq_error:
 }
 
 static inline int
-nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_stop_rx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+		       uint16_t qidx)
 {
-	struct nicvf *nic = nicvf_pmd_priv(dev);
 	struct nicvf_rxq *rxq;
 	int ret, other_error;
 
-	if (dev->data->rx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STOPPED)
+	if (dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+		RTE_ETH_QUEUE_STATE_STOPPED)
 		return 0;
 
 	ret = nicvf_qset_rq_reclaim(nic, qidx);
 	if (ret)
-		PMD_INIT_LOG(ERR, "Failed to reclaim rq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to reclaim rq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 
 	other_error = ret;
-	rxq = dev->data->rx_queues[qidx];
+	rxq = dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)];
 	nicvf_rx_queue_release_mbufs(dev, rxq);
 	nicvf_rx_queue_reset(rxq);
 
 	ret = nicvf_qset_cq_reclaim(nic, qidx);
 	if (ret)
-		PMD_INIT_LOG(ERR, "Failed to reclaim cq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to reclaim cq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 
 	other_error |= ret;
-	dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+	dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STOPPED;
 	return other_error;
 }
 
@@ -1057,8 +1071,14 @@ static int
 nicvf_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
 {
 	int ret;
+	struct nicvf *nic = nicvf_pmd_priv(dev);
 
-	ret = nicvf_stop_rx_queue(dev, qidx);
+	if (qidx >= MAX_SND_QUEUES_PER_QS)
+		nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+	qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
+	ret = nicvf_vf_stop_rx_queue(dev, nic, qidx);
 	ret |= nicvf_configure_cpi(dev);
 	ret |= nicvf_configure_rss_reta(dev);
 	return ret;
@@ -1073,7 +1093,14 @@ nicvf_dev_tx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
 static int
 nicvf_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
 {
-	return nicvf_stop_tx_queue(dev, qidx);
+	struct nicvf *nic = nicvf_pmd_priv(dev);
+
+	if (qidx >= MAX_SND_QUEUES_PER_QS)
+		nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+	qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
+	return nicvf_vf_stop_tx_queue(dev, nic, qidx);
 }
 
 static int
@@ -1442,10 +1469,10 @@ qset_rss_error:
 	nicvf_rss_term(nic);
 start_txq_error:
 	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++)
-		nicvf_stop_tx_queue(dev, qidx);
+		nicvf_vf_stop_tx_queue(dev, nic, qidx);
 start_rxq_error:
 	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++)
-		nicvf_stop_rx_queue(dev, qidx);
+		nicvf_vf_stop_rx_queue(dev, nic, qidx);
 qset_rbdr_reclaim:
 	nicvf_qset_rbdr_reclaim(nic, 0);
 	nicvf_rbdr_release_mbufs(dev, nic);
@@ -1460,32 +1487,74 @@ qset_reclaim:
 }
 
 static void
-nicvf_dev_stop(struct rte_eth_dev *dev)
+nicvf_dev_stop_cleanup(struct rte_eth_dev *dev, bool cleanup)
 {
+	size_t i;
 	int ret;
-	uint16_t qidx;
 	struct nicvf *nic = nicvf_pmd_priv(dev);
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Let PF make the BGX's RX and TX switches to OFF position */
-	nicvf_mbox_shutdown(nic);
+	/* Teardown secondary vf first */
+	for (i = 0; i < nic->sqs_count; i++) {
+		if (!nic->snicvf[i])
+			continue;
+
+		nicvf_vf_stop(dev, nic->snicvf[i], cleanup);
+	}
+
+	/* Stop the primary VF now */
+	nicvf_vf_stop(dev, nic, cleanup);
 
 	/* Disable loopback */
 	ret = nicvf_loopback_config(nic, 0);
 	if (ret)
 		PMD_INIT_LOG(ERR, "Failed to disable loopback %d", ret);
 
+	/* Reclaim CPI configuration */
+	ret = nicvf_mbox_config_cpi(nic, 0);
+	if (ret)
+		PMD_INIT_LOG(ERR, "Failed to reclaim CPI config %d", ret);
+}
+
+static void
+nicvf_dev_stop(struct rte_eth_dev *dev)
+{
+	PMD_INIT_FUNC_TRACE();
+
+	nicvf_dev_stop_cleanup(dev, false);
+}
+
+static void
+nicvf_vf_stop(struct rte_eth_dev *dev, struct nicvf *nic, bool cleanup)
+{
+	int ret;
+	uint16_t qidx;
+	uint16_t tx_start, tx_end;
+	uint16_t rx_start, rx_end;
+
+	PMD_INIT_FUNC_TRACE();
+
+	if (cleanup) {
+		/* Let PF make the BGX's RX and TX switches to OFF position */
+		nicvf_mbox_shutdown(nic);
+	}
+
 	/* Disable VLAN Strip */
 	nicvf_vlan_hw_strip(nic, 0);
 
-	/* Reclaim sq */
-	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++)
-		nicvf_stop_tx_queue(dev, qidx);
+	/* Get queue ranges for this VF */
+	nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+
+	for (qidx = tx_start; qidx <= tx_end; qidx++)
+		nicvf_vf_stop_tx_queue(dev, nic, qidx % MAX_SND_QUEUES_PER_QS);
+
+	/* Get queue ranges for this VF */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
 
 	/* Reclaim rq */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++)
-		nicvf_stop_rx_queue(dev, qidx);
+	for (qidx = rx_start; qidx <= rx_end; qidx++)
+		nicvf_vf_stop_rx_queue(dev, nic, qidx % MAX_RCV_QUEUES_PER_QS);
 
 	/* Reclaim RBDR */
 	ret = nicvf_qset_rbdr_reclaim(nic, 0);
@@ -1496,15 +1565,8 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
 	if (nic->rbdr != NULL)
 		nicvf_rbdr_release_mbufs(dev, nic);
 
-	/* Reclaim CPI configuration */
-	if (!nic->sqs_mode) {
-		ret = nicvf_mbox_config_cpi(nic, 0);
-		if (ret)
-			PMD_INIT_LOG(ERR, "Failed to reclaim CPI config");
-	}
-
 	/* Disable qset */
-	ret = nicvf_qset_config(nic);
+	ret = nicvf_qset_reclaim(nic);
 	if (ret)
 		PMD_INIT_LOG(ERR, "Failed to disable qset %d", ret);
 
@@ -1521,10 +1583,20 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
 static void
 nicvf_dev_close(struct rte_eth_dev *dev)
 {
+	size_t i;
+	struct nicvf *nic = nicvf_pmd_priv(dev);
+
 	PMD_INIT_FUNC_TRACE();
 
-	nicvf_dev_stop(dev);
+	nicvf_dev_stop_cleanup(dev, true);
 	nicvf_periodic_alarm_stop(nicvf_interrupt, dev);
+
+	for (i = 0; i < nic->sqs_count; i++) {
+		if (!nic->snicvf[i])
+			continue;
+
+		nicvf_periodic_alarm_stop(nicvf_vf_interrupt, nic->snicvf[i]);
+	}
 }
 
 static int
-- 
1.9.1