From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-co1nam03on0047.outbound.protection.outlook.com [104.47.40.47]) by dpdk.org (Postfix) with ESMTP id CAB356947 for ; Fri, 26 Aug 2016 18:55:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=/z3FtVwQx4uCIVXLAup1CS2Tl9P7afv/0w3vz57k6SU=; b=U3uFGRc4GyngCi6LexdSvf/TXwg5a2qTdYujS/ULktiz0S7QOMDkO+EBhKjJ7o+mm/ox5waLCoRFTvFLtYKX8iB7s8PxNGSWweGPJHOcg4W6WTo087yyVYxbniv2mwIQEPuXigPzzNUgHsjlzdRRHtispXhvdVOXVgFHIDNZUwQ= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Kamil.Rytarowski@cavium.com; Received: from cavium1.semihalf.local (31.172.191.173) by CY4PR07MB3064.namprd07.prod.outlook.com (10.172.116.145) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.549.15; Fri, 26 Aug 2016 16:55:17 +0000 From: Kamil Rytarowski To: CC: , , , , , Kamil Rytarowski Date: Fri, 26 Aug 2016 18:54:07 +0200 Message-ID: <1472230448-17490-13-git-send-email-krytarowski@caviumnetworks.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1472230448-17490-1-git-send-email-krytarowski@caviumnetworks.com> References: <1472230448-17490-1-git-send-email-krytarowski@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [31.172.191.173] X-ClientProxiedBy: HE1PR0201CA0008.eurprd02.prod.outlook.com (10.165.229.18) To CY4PR07MB3064.namprd07.prod.outlook.com (10.172.116.145) X-MS-Office365-Filtering-Correlation-Id: bb7b89b5-8cb3-4b0a-816a-08d3cdd1c32c X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064; 2:6SvruZ81lzpjqngMb+k2eG8TbN9Ll9prXWGv2UeXw2yo/OBAwTjP58hGIecpzpn/RKgfvQjklQcICjnpEdVOC0yHC0lLsuK/g1KBRAsBOLOOZBAwgU+WBYC9r5CCbaRUDnP4ZtKUjXbqsRrfxh2z4NV3ankWodDykgCasdAn4PZwx9g6P1mOf1opY1KjDy9Q; 3:XdVED8kR6sHobfGtqXNMivRUMLmVr7VYP+YvdDdsNzMEgGH+cRh0O4hcoAyRsZ8R/E2BmZ4oabeHr2GSTn3TCMeB08TzIf7XHCfw+j14aYxbUtHee985TV4egfoJysZz X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY4PR07MB3064; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064; 25:3sizQLm/IxVSTpp/CsNzewlKF3ChjBp0l9s9Nxy4ESMKl297hAlzg8I2w2TQJJlHWPiLcBf5mrhLKavdZFxD1r18Te5FhmW18CbmYx6Lq/M3BiRV49jh+qPoPisanww2v4W9Wz8Jkcc+iYySNRmBc/DOnbZshvQmUTSuNGJoNaAiSYsCumxyRUZ67g3QGUQ8wFrVqYBAomSU5y2dpEelq/E6+CD++N+3eIqbnxUVN3g8hzypJOlFDKqXQCs6es4FXwDcMmoTvQHMl5Lqwx5+xpgjNDwZJwRBU17Sh8gL3rvuzg0nT94zqOkOKNIL7whEg7slVYc9nYT2Ef5mb1rbyqDtPvTGxL3G0gI7TszN5FmiqaPdoXzMhFePhlLeNfdzJsIu1nS3uTIsJQjHHblzdJC/aCQ/xdDGgI3AHyT5pB8IINZDF62gtQyeoaH1CnawDQGR8XqaSThOVold+jE11Z41k60LrB4HTrbWOCa/fYFnxGB94Yk9oMR2P+vRGFF0b6zjF8CecUvT4Qc/21UVXlJkTf4Oeliw+KjB9y85to6A8VHSRmGKmAfE4m0jTsKpkrG+d2c/PjncojrlRPEXkgtkGhjqp8bd2+ZCFHzOr45DukZozlKUph3cKjAkNo7fcdRPxkiWdvfb/wpYtAcHxolz6HeICIvMgQXFem3KmU2ohgEIzz6GrA4meCMLkrmT6TlLmxop5mcSiH0rS4Cky7xZFqkW5HBGGuGI5e45484= X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064; 31:s0BLlM8fjxrLwgpHtGF8EVVj9egIUj2uolh7ukjPR/yo05C6nVUbW7JKci2INYqZbE0gT6eGwmqNeShFlCYl/xHWFXp+AKG7chDGhRaL7PQOKYjYMmgUFKp2/UPa/hG+znJyvaqaLwjl5T0E/chtHx/plaEZB3ZKCokhaL+6cB7Au3yMHyr3CbbKIWdTljskJb3zXim3ZTv5GT7TC0aBIaDyxHwTcIO/oIOWFWZpDvY=; 20:fLZQUk9iYcmL6sLVG8cT9JJQHG7St+1P92YyoprInyFu/9uHV1nsb968Y9/80fZUs6sA8ZvJbdd603hXtHH0qn7boihMINDO2UAf80Veyb/712XCJMl3zIhM/zIX2YdI/HZnhB9z9vnvwnmHXThKUkBU8cj9DalpMFTLn7jvbyPlR9F0iIgbErQ4mC7CKe3OA3f7uhaCmBPiMnQUffb/9zQZqGwuGdL6N1DxB5kFmWN0n2immkU9MZ7CM+fUC72ZMXnjwgv7Kb8n907u9dXKHqG5QpVgQhhnmXa/Jh73KTZ81llkRv0797vRWxil4H3R+7TFZNoI09pFOvFT4gZsD4o6z7g08eYHv5tejxOZGj517DmXsZHQsZ7IZ+SFASeomKJaXKGMe4/6I6Vz/0q0j+PAGOx9m5M/AS6jw1NYyzemCj/K9w27FoZnNf61WJ/U7uBJpa0RyyLSYgH53ZZLcjsaFf9/QNacuojqOPj5L3O5pVlDCWuG+LlbTOj4ds1lvCU/493Q+5LtaIq1HvoTCXF0QtI0MLQ38Nt6Z8EUo53gSWjnBpx2uxq3mKiwryTWdPagIczZih99W5PRvH44+RNlFCEkWZAyu+VB+XdyWCo= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001); SRVR:CY4PR07MB3064; BCL:0; PCL:0; RULEID:; SRVR:CY4PR07MB3064; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064; 4:XRr24UOxhfh5dpnmD+ij08TNo+N4R8lanQhKvXrGoP2JGJVPebnJh/XZND33GkF29e+awLmNnbwb0xtGcPVJ44EiXUjtLcSlEpyPEy1x6IHYd8ukoxxc5INMJdcw8aP3ERuMJtDFmFPQjLrMLyCY/PpO1TtvM23D9JQLD5aNuXIrQvq0G9S4PfYzpP2t/dvvZf+2XMsSBdinx0uoELXbtMfqOyqSdoL7YTi/VRppRsxC2gtP3LmjjLcz8qyDLwKufEgCkEAfpP9IV/rQ/JHFMpd/YYb3UvwubU87LsgGqFYMIxklDBB7LoSU2QqSq+y71P2ASIhovKIfvGkl5lQzq8e0fdOpQE8jr0W/rRnJZrx5k8Pe4wOf+nFklmewnv3YnbbtrSpgHmkyYP36/fbVzg== X-Forefront-PRVS: 00462943DE X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(7916002)(189002)(199003)(81156014)(36756003)(77096005)(189998001)(2906002)(229853001)(2351001)(105586002)(8676002)(101416001)(92566002)(5660300001)(81166006)(6116002)(4001430100002)(2950100001)(7736002)(47776003)(4326007)(66066001)(586003)(50466002)(106356001)(50226002)(5003940100001)(19580405001)(19580395003)(305945005)(50986999)(110136002)(76176999)(7846002)(48376002)(68736007)(42186005)(3846002)(107886002)(33646002)(97736004); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR07MB3064; H:cavium1.semihalf.local; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR07MB3064; 23:5zIe+x03FjHCib2hk1DRg7HPgsA0QqIJ/9f5jTflj?= =?us-ascii?Q?4RsUL+aycrniIU/Fs1/vWRbWMtJoadU98p9JuaUUw1MiTk6UrQySoTgFaPxD?= =?us-ascii?Q?AQ8R/INEwjrH2FmAWxOelw8w19ajGh+qneFrp25vmKgyqm6oF8wiHLe7J7Q0?= =?us-ascii?Q?PkCNuMQ2cgWxEiHgrn/Hj+iIk4haeO/mY8wnehUZ30sy16Uy9Q7wK4nrymJo?= =?us-ascii?Q?SveIoAPjIWkdb2P/ODlHXDzCF0pwZPqZRaFqxNxo0gQkITCOme17YercEG21?= =?us-ascii?Q?zzC8KLXc8wQJlwODNPDg0ve2cyrxIOlvAGAve27GrhGNDCX8jdoNHEcz5bfa?= =?us-ascii?Q?j/wWC+cnltupYB4zaYu8Pif2iE038MMoPXOHQExQV9UY4ea8rZqsQSPJLkqb?= =?us-ascii?Q?BLRXZY8FoKTW6MBih4oQzz+RoocfD9p6t2NCzPOQAznNPwX8WrtJPj+WK8Cp?= =?us-ascii?Q?j3oK818BCG4cTehkRsbAq6KFgjSh9jEQq9Pe4MEitxqtMTaJbniwQUI/tPzT?= =?us-ascii?Q?XnkMnrCDmm0EXs2GQhMv7MIEV69FYTNKw249LvlVv3p4MLoLuZ17mNlQod6W?= =?us-ascii?Q?AzXvICg/4xdM16jbp04NFwqCn1fma6jLewKjBSmcQ9P0TKxqrvuYcT2c5V07?= =?us-ascii?Q?UVIu+aQnJ18EJxqDvLNzmmCZBvebo/WU7uM8lwQxXroPINesXtqqGJNfU84i?= =?us-ascii?Q?7a378poXBxQ6lgqAZeaARdpCU0KsyegrIJRyb3V6V8PLnzIDKFJvYdPTI7FB?= =?us-ascii?Q?fj/+pXoN8wn+OVyj9+nlHS2IwLXOcS6mPTiS883hdKabZ+dNvFIUV0B0xEI6?= =?us-ascii?Q?x5DjiZQq/E2Wcc5NAJFIKHdgGjItDipgNcLEmRPwopZKRSef7pnxJpBK1cZy?= =?us-ascii?Q?YfAtyM8gFqHFP9XLiC/o+n23Cnud35L7bJEpxhTbbeuzNtUXMYfdjpkHwK5V?= =?us-ascii?Q?vcxg/KIcB/lN2hH2C411XksxNai7eXumUWg4AG8mWfzsCnSKE4mhOpO2Lpau?= =?us-ascii?Q?7/TcBKflqqu3jfy1Ad48HhsC+OHLt8vXTFVRn+KOLGeUkL3uaSb/4Vmf3byZ?= =?us-ascii?Q?lYJermnPfkRxaUG/uWJHxPbH1niVZ5wkjeyHmMI6B4dAMKyUpMUICReIDHSg?= =?us-ascii?Q?SsemHhY6wQ=3D?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064; 6:EBRTfth35TaP40ibXfdE4wqhdW+ZRCYmTo+1HfdddMJpd9LvJf0V76fhtaLK+zNdw+P3N6hM3oBx1hpFhSZkJ0HlYXYiad8p7ioiVOg4SaVDIUJdj4U1ZfCt2TH4OK4w+LjExz9OmSdfXXJuiPkkgIRjhbgG5vqyaxORZ1Ygmlnoa1ssX64fHj+31Wt9FgtxwnnN/Hl8Mi2qAF39BQdKBq0EOVZk+yLQSmd/7+6P8P+L4DjXDVdTisqf5rtZPf3GyCIzhHhb8ehiy3RajLmDV+Iv/Y3L2/cLJWHI0HOdN+U=; 5:K8a2FExNAw127YcNB7Y1eXtLL1SdLLhVS7FWH8EycXvV+JVwQWZd3GbgykggTe6zq/qWSMAUb7m65rrueaK+DZKsixuUDunDg43VB4nbk9uswgl4iRsGqqnq+1W5hJDQfp+pcpfhqmgNTHU09aa1vg==; 24:ezSBuzSt2bblIbk9gT9v2FkVh7TopL2b/5e2m4ghHkCmbwHdil5T/ywH0OqPvTNwAcV0iTaUSfyIWBjANM/2LSbrXlUvWmcr/ANMHqQTfg4=; 7:6/yPLoOzKlxxqC2EHEqXEP1tbn/JPA7gftazcQjKJX7ly6ZjZ/8MO+puiIcA2poCYmsN5kfO2mH5OLdJHQoKO4UvPhdINacEN4IUGaHEITfd0rcEUnYG1dRyhine2oNwRx2Sjhsu/nb2TmzpbTBxhC+dQ5stNH9Ckvm1YEPm40Z3LuAAZPTQ/IgacndGAKaB9IKJq3lRMsCxXIFVV0s/50DLb8HB2MeQEmVaaFSzE1DstJRmoKRaDXFkPxn/1HOu SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2016 16:55:17.7589 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR07MB3064 Subject: [dpdk-dev] [PATCH 12/13] net/thunderx: add final bits for secondary queue support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Aug 2016 16:55:23 -0000 From: Kamil Rytarowski Signed-off-by: Maciej Czekaj Signed-off-by: Kamil Rytarowski Signed-off-by: Zyta Szpak Signed-off-by: Slawomir Rosek Signed-off-by: Radoslaw Biernacki Signed-off-by: Jerin Jacob --- drivers/net/thunderx/nicvf_ethdev.c | 178 +++++++++++++++++++++++++++++------- 1 file changed, 144 insertions(+), 34 deletions(-) diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c index fcdbebf..04bfde0 100644 --- a/drivers/net/thunderx/nicvf_ethdev.c +++ b/drivers/net/thunderx/nicvf_ethdev.c @@ -118,7 +118,7 @@ nicvf_interrupt(void *arg) nicvf_interrupt, dev); } -static void __rte_unused +static void nicvf_vf_interrupt(void *arg) { struct nicvf *nic = arg; @@ -236,9 +236,15 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) struct nicvf_hw_tx_qstats tx_qstats; struct nicvf_hw_stats port_stats; struct nicvf *nic = nicvf_pmd_priv(dev); + uint16_t rx_start, rx_end; + uint16_t tx_start, tx_end; + size_t i; + + /* RX queue indices for the first VF */ + nicvf_rx_range(dev, nic, &rx_start, &rx_end); /* Reading per RX ring stats */ - for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) { + for (qidx = rx_start; qidx <= rx_end; qidx++) { if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS) break; @@ -247,8 +253,11 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) stats->q_ipackets[qidx] = rx_qstats.q_rx_packets; } + /* TX queue indices for the first VF */ + nicvf_tx_range(dev, nic, &tx_start, &tx_end); + /* Reading per TX ring stats */ - for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++) { + for (qidx = tx_start; qidx <= tx_end; qidx++) { if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS) break; @@ -257,6 +266,40 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) stats->q_opackets[qidx] = tx_qstats.q_tx_packets; } + for (i = 0; i < nic->sqs_count; i++) { + struct nicvf *snic = nic->snicvf[i]; + + if (snic == NULL) + break; + + /* RX queue indices for a secondary VF */ + nicvf_rx_range(dev, snic, &rx_start, &rx_end); + + /* Reading per RX ring stats */ + for (qidx = rx_start; qidx <= rx_end; qidx++) { + if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS) + break; + + nicvf_hw_get_rx_qstats(snic, &rx_qstats, + qidx % MAX_RCV_QUEUES_PER_QS); + stats->q_ibytes[qidx] = rx_qstats.q_rx_bytes; + stats->q_ipackets[qidx] = rx_qstats.q_rx_packets; + } + + /* TX queue indices for a secondary VF */ + nicvf_tx_range(dev, snic, &tx_start, &tx_end); + /* Reading per TX ring stats */ + for (qidx = tx_start; qidx <= tx_end; qidx++) { + if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS) + break; + + nicvf_hw_get_tx_qstats(snic, &tx_qstats, + qidx % MAX_SND_QUEUES_PER_QS); + stats->q_obytes[qidx] = tx_qstats.q_tx_bytes; + stats->q_opackets[qidx] = tx_qstats.q_tx_packets; + } + } + nicvf_hw_get_stats(nic, &port_stats); stats->ibytes = port_stats.rx_bytes; stats->ipackets = port_stats.rx_ucast_frames; @@ -322,13 +365,36 @@ nicvf_dev_stats_reset(struct rte_eth_dev *dev) int i; uint16_t rxqs = 0, txqs = 0; struct nicvf *nic = nicvf_pmd_priv(dev); + uint16_t rx_start, rx_end; + uint16_t tx_start, tx_end; - for (i = 0; i < dev->data->nb_rx_queues; i++) + /* Reset all primary nic counters */ + nicvf_rx_range(dev, nic, &rx_start, &rx_end); + for (i = rx_start; i <= rx_end; i++) rxqs |= (0x3 << (i * 2)); - for (i = 0; i < dev->data->nb_tx_queues; i++) + + nicvf_tx_range(dev, nic, &tx_start, &tx_end); + for (i = tx_start; i <= tx_end; i++) txqs |= (0x3 << (i * 2)); nicvf_mbox_reset_stat_counters(nic, 0x3FFF, 0x1F, rxqs, txqs); + + /* Reset secondary nic queue counters */ + for (i = 0; i < nic->sqs_count; i++) { + struct nicvf *snic = nic->snicvf[i]; + if (snic == NULL) + break; + + nicvf_rx_range(dev, snic, &rx_start, &rx_end); + for (i = rx_start; i <= rx_end; i++) + rxqs |= (0x3 << ((i % MAX_CMP_QUEUES_PER_QS) * 2)); + + nicvf_tx_range(dev, snic, &tx_start, &tx_end); + for (i = tx_start; i <= tx_end; i++) + txqs |= (0x3 << ((i % MAX_SND_QUEUES_PER_QS) * 2)); + + nicvf_mbox_reset_stat_counters(snic, 0, 0, rxqs, txqs); + } } /* Promiscuous mode enabled by default in LMAC to VF 1:1 map configuration */ @@ -596,14 +662,18 @@ nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf *nic, } static void -nicvf_rbdr_release_mbuf(struct rte_eth_dev *dev, struct nicvf *nic __rte_unused, +nicvf_rbdr_release_mbuf(struct rte_eth_dev *dev, struct nicvf *nic, nicvf_phys_addr_t phy) { uint16_t qidx; void *obj; struct nicvf_rxq *rxq; + uint16_t rx_start, rx_end; - for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) { + /* Get queue ranges for this VF */ + nicvf_rx_range(dev, nic, &rx_start, &rx_end); + + for (qidx = rx_start; qidx <= rx_end; qidx++) { rxq = dev->data->rx_queues[qidx]; if (rxq->precharge_cnt) { obj = (void *)nicvf_mbuff_phy2virt(phy, @@ -861,6 +931,11 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx, PMD_INIT_FUNC_TRACE(); + if (qidx >= MAX_SND_QUEUES_PER_QS) + nic = nic->snicvf[qidx / MAX_SND_QUEUES_PER_QS - 1]; + + qidx = qidx % MAX_SND_QUEUES_PER_QS; + /* Socket id check */ if (socket_id != (unsigned int)SOCKET_ID_ANY && socket_id != nic->node) PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d", @@ -895,18 +970,20 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx, } /* Free memory prior to re-allocation if needed. */ - if (dev->data->tx_queues[qidx] != NULL) { + if (dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)] != NULL) { PMD_TX_LOG(DEBUG, "Freeing memory prior to re-allocation %d", - qidx); - nicvf_dev_tx_queue_release(dev->data->tx_queues[qidx]); - dev->data->tx_queues[qidx] = NULL; + nicvf_netdev_qidx(nic, qidx)); + nicvf_dev_tx_queue_release( + dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)]); + dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)] = NULL; } /* Allocating tx queue data structure */ txq = rte_zmalloc_socket("ethdev TX queue", sizeof(struct nicvf_txq), RTE_CACHE_LINE_SIZE, nic->node); if (txq == NULL) { - PMD_INIT_LOG(ERR, "Failed to allocate txq=%d", qidx); + PMD_INIT_LOG(ERR, "Failed to allocate txq=%d", + nicvf_netdev_qidx(nic, qidx)); return -ENOMEM; } @@ -949,10 +1026,12 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx, nicvf_tx_queue_reset(txq); PMD_TX_LOG(DEBUG, "[%d] txq=%p nb_desc=%d desc=%p phys=0x%" PRIx64, - qidx, txq, nb_desc, txq->desc, txq->phys); + nicvf_netdev_qidx(nic, qidx), txq, nb_desc, txq->desc, + txq->phys); - dev->data->tx_queues[qidx] = txq; - dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED; + dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)] = txq; + dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] = + RTE_ETH_QUEUE_STATE_STOPPED; return 0; } @@ -967,7 +1046,8 @@ nicvf_rx_queue_release_mbufs(struct rte_eth_dev *dev, struct nicvf_rxq *rxq) if (dev->rx_pkt_burst == NULL) return; - while ((rxq_cnt = nicvf_dev_rx_queue_count(dev, rxq->queue_id))) { + while ((rxq_cnt = nicvf_dev_rx_queue_count(dev, + nicvf_netdev_qidx(rxq->nic, rxq->queue_id)))) { nb_pkts = dev->rx_pkt_burst(rxq, rx_pkts, NICVF_MAX_RX_FREE_THRESH); PMD_DRV_LOG(INFO, "nb_pkts=%d rxq_cnt=%d", nb_pkts, rxq_cnt); @@ -977,7 +1057,10 @@ nicvf_rx_queue_release_mbufs(struct rte_eth_dev *dev, struct nicvf_rxq *rxq) } } - refill_cnt += nicvf_dev_rbdr_refill(dev, rxq->queue_id); + + refill_cnt += nicvf_dev_rbdr_refill(dev, + nicvf_netdev_qidx(rxq->nic, rxq->queue_id)); + PMD_DRV_LOG(INFO, "free_cnt=%d refill_cnt=%d", released_pkts, refill_cnt); } @@ -1148,6 +1231,11 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx, PMD_INIT_FUNC_TRACE(); + if (qidx >= MAX_RCV_QUEUES_PER_QS) + nic = nic->snicvf[qidx / MAX_RCV_QUEUES_PER_QS - 1]; + + qidx = qidx % MAX_RCV_QUEUES_PER_QS; + /* Socket id check */ if (socket_id != (unsigned int)SOCKET_ID_ANY && socket_id != nic->node) PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d", @@ -1184,18 +1272,20 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx, } /* Free memory prior to re-allocation if needed */ - if (dev->data->rx_queues[qidx] != NULL) { + if (dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)] != NULL) { PMD_RX_LOG(DEBUG, "Freeing memory prior to re-allocation %d", - qidx); - nicvf_dev_rx_queue_release(dev->data->rx_queues[qidx]); - dev->data->rx_queues[qidx] = NULL; + nicvf_netdev_qidx(nic, qidx)); + nicvf_dev_rx_queue_release( + dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)]); + dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)] = NULL; } /* Allocate rxq memory */ rxq = rte_zmalloc_socket("ethdev rx queue", sizeof(struct nicvf_rxq), RTE_CACHE_LINE_SIZE, nic->node); if (rxq == NULL) { - PMD_INIT_LOG(ERR, "Failed to allocate rxq=%d", qidx); + PMD_INIT_LOG(ERR, "Failed to allocate rxq=%d", + nicvf_netdev_qidx(nic, qidx)); return -ENOMEM; } @@ -1225,11 +1315,12 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx, nicvf_rx_queue_reset(rxq); PMD_RX_LOG(DEBUG, "[%d] rxq=%p pool=%s nb_desc=(%d/%d) phy=%" PRIx64, - qidx, rxq, mp->name, nb_desc, + nicvf_netdev_qidx(nic, qidx), rxq, mp->name, nb_desc, rte_mempool_avail_count(mp), rxq->phys); - dev->data->rx_queues[qidx] = rxq; - dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED; + dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)] = rxq; + dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] = + RTE_ETH_QUEUE_STATE_STOPPED; return 0; } @@ -1242,8 +1333,10 @@ nicvf_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) dev_info->min_rx_bufsize = ETHER_MIN_MTU; dev_info->max_rx_pktlen = NIC_HW_MAX_FRS; - dev_info->max_rx_queues = (uint16_t)MAX_RCV_QUEUES_PER_QS; - dev_info->max_tx_queues = (uint16_t)MAX_SND_QUEUES_PER_QS; + dev_info->max_rx_queues = + (uint16_t)MAX_RCV_QUEUES_PER_QS * (MAX_SQS_PER_VF + 1); + dev_info->max_tx_queues = + (uint16_t)MAX_SND_QUEUES_PER_QS * (MAX_SQS_PER_VF + 1); dev_info->max_mac_addrs = 1; dev_info->max_vfs = dev->pci_dev->max_vfs; @@ -1284,9 +1377,13 @@ rbdr_rte_mempool_get(void *dev, void *opaque) uintptr_t mbuf; struct nicvf_rxq *rxq; struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)dev; - struct nicvf *nic __rte_unused = (struct nicvf *)opaque; + struct nicvf *nic = (struct nicvf *)opaque; + uint16_t rx_start, rx_end; + + /* Get queue ranges for this VF */ + nicvf_rx_range(eth_dev, nic, &rx_start, &rx_end); - for (qidx = 0; qidx < eth_dev->data->nb_rx_queues; qidx++) { + for (qidx = rx_start; qidx <= rx_end; qidx++) { rxq = eth_dev->data->rx_queues[qidx]; /* Maintain equal buffer count across all pools */ if (rxq->precharge_cnt >= rxq->qlen_mask) @@ -1852,7 +1949,9 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev) /* Setup callbacks for secondary process */ nicvf_set_tx_function(eth_dev); nicvf_set_rx_function(eth_dev); - return 0; + /* If nic == NULL than it is secondary function + * so ethdev need to be released by caller */ + return nic != NULL ? 0 : ENOTSUP; } pci_dev = eth_dev->pci_dev; @@ -1898,10 +1997,21 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev) } if (nic->sqs_mode) { - PMD_INIT_LOG(INFO, "Unsupported SQS VF detected, Detaching..."); - /* Detach port by returning Positive error number */ - ret = ENOTSUP; - goto alarm_fail; + /* Push nic to stack of secondary vfs */ + nicvf_svf_push(nic); + + /* Steal nic pointer from the device for further reuse */ + eth_dev->data->dev_private = NULL; + + nicvf_periodic_alarm_stop(nicvf_interrupt, eth_dev); + ret = nicvf_periodic_alarm_start(nicvf_vf_interrupt, nic); + if (ret) { + PMD_INIT_LOG(ERR, "Failed to start period alarm"); + goto fail; + } + + /* Detach port by returning postive error number */ + return ENOTSUP; } eth_dev->data->mac_addrs = rte_zmalloc("mac_addr", ETHER_ADDR_LEN, 0); -- 1.9.1