From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR03-DB5-obe.outbound.protection.outlook.com (mail-eopbgr40085.outbound.protection.outlook.com [40.107.4.85]) by dpdk.org (Postfix) with ESMTP id 024C6201 for ; Thu, 1 Nov 2018 08:21:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eLptmWH/zNZPK6EeA12hprJFUaksHOIhPn+P5mJ6fCA=; b=IJSlxWop5425CqyNytWZlT4EA3zERhAUWoj0KuAuCJRZljXOHRINWIOaQbvSrUmaMuXtulvVGamhYUojoHl3g+Viv0kyvTgv1R+ORiFkiUDs4ddX0ApQHPAciQiJ3YA/3M3Hhx4dPibwMfXKhAt58weUiVfw/jkZtKT4qIdbKew= Received: from DB7PR05MB4426.eurprd05.prod.outlook.com (52.134.109.15) by DB7PR05MB4107.eurprd05.prod.outlook.com (52.134.107.32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1294.22; Thu, 1 Nov 2018 07:21:32 +0000 Received: from DB7PR05MB4426.eurprd05.prod.outlook.com ([fe80::80e:e6b:baf2:d973]) by DB7PR05MB4426.eurprd05.prod.outlook.com ([fe80::80e:e6b:baf2:d973%3]) with mapi id 15.20.1273.027; Thu, 1 Nov 2018 07:21:32 +0000 From: Shahaf Shuler To: Tom Barbette , "dev@dpdk.org" , Ferruh Yigit , Thomas Monjalon , Andrew Rybchenko , "olivier.matz@6wind.com" CC: Yongseok Koh Thread-Topic: [PATCH v4] mlx5: Support for rte_eth_rx_queue_count Thread-Index: AQHUbgdFa7YpZOBUAEWIgl1+xlkwxaU0YpJwgASx7wCAAXMSEA== Date: Thu, 1 Nov 2018 07:21:31 +0000 Message-ID: References: <1540653055-67051-1-git-send-email-barbette@kth.se>, <1540976475938.69727@kth.se> In-Reply-To: <1540976475938.69727@kth.se> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB7PR05MB4107; 6:w2ycGFVxxD44rTXxNJEmcNK7EjqYwgEdxo5Voos2rt+MDv9iNqTrpa9JXgcQkcixrwdqPZwoflioJ/NhXoiNzsPVxCLjxRoBhzRWLWs6ajVRlZNebXqmuLBSn1xQQuElxQJwK+v3/TH/tFHxbPWAkHscwuaS0eHaXbi8eAVSbH5r/+wtTS2ExMaGUraWsgIBkB5uuOCOVMpsirbeCabj+iu69+frEqgW9q18n/kgbeT5zj2X82BatjHVOtq4LLHpCqC/7VacrxI61AXgDp4Gi30WWmkMsCccfGfoO7QysV7ILXGT9DS70pUQ0lsOQZTB+ZlQ7J+/DE/0fuZDktLpGQOzzoKV+HAiACbh8elj15kl5u0b0I7HSIEw7+A1CGwLwiIwZ+nxG8INnr1ycwlJE6uk+fiM1BO/FV8T+aEQ4ZR17z2myOY1FKPIgVZ1d3dLtd3nVxRM8mx3RAktGhcCSg==; 5:0rYgkPcU+4N0rzk5/cwEVD9PNN1jb2iGLz/qXBxIrse3YUH5z9v/D/yuJWSqgRwiX2mFsMQau+m/l7orEFluwoCVlAP1EZD5FdlVhzIUCdhEF6I8uKUTN/EOMAsm3BVM62EfDHjlZY2AYAh2Rb2LX0EhXqcwCHKoxXTQVrulXxE=; 7:qgWctuFhDil5+trlXVrfd9FxPCPAb+SiMiw5z0mbd+t5Iv4+G48F1C6sASRLjVaP6NOJ549QXGe9BOTPbjkKneeYbLJtWSgv56q7bU+/uxQ0gI5x5cK2VYUtK3bEQyzMoDfDiy2902rZq0Ppc9+33w== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: ee42da05-bd4e-4054-cf4b-08d63fcaa5ff x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB7PR05MB4107; x-ms-traffictypediagnostic: DB7PR05MB4107: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(3231382)(944501410)(52105095)(10201501046)(3002001)(6055026)(148016)(149066)(150057)(6041310)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123564045)(20161123562045)(201708071742011)(7699051)(76991095); SRVR:DB7PR05MB4107; BCL:0; PCL:0; RULEID:; SRVR:DB7PR05MB4107; x-forefront-prvs: 0843C17679 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(979002)(376002)(366004)(346002)(396003)(39860400002)(136003)(199004)(189003)(9686003)(71190400001)(55016002)(316002)(8936002)(26005)(5660300001)(66066001)(74316002)(110136005)(5250100002)(105586002)(71200400001)(81166006)(6436002)(2906002)(97736004)(68736007)(86362001)(102836004)(8676002)(6116002)(575784001)(81156014)(3846002)(2501003)(106356001)(7696005)(6506007)(76176011)(99286004)(107886003)(478600001)(33656002)(229853002)(486006)(25786009)(14454004)(4326008)(53936002)(6246003)(7736002)(305945005)(2900100001)(186003)(11346002)(446003)(256004)(476003)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB7PR05MB4107; H:DB7PR05MB4426.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: vSVgJ8YodkBilghi5ImJUBa2R4Y/qBnEtLoKh5Bvv45DR55beCk5ljzoIzYv2sYjZ86h2VRzoVhDWDWcvJJ8KHJgYRDS8DghZ+oY2bGGYFsbTJq6Gl+iHBUgDt9esy+OBAvzAbez6M8FbnTIhmJzNSNQ5ges0uoTEo7TPzLgE88coNBcscW/lOn/XT2OVTmHLWvOfR2ni8iGm5N5kekcKdK6Jcxrh29+juEJemb6y1lwvQbVElqFIDKQYq1ok0b4vX4YAbZsnZqGYJdPcycL4G8lbF+EvnD4M90sMa/4KA/SISUMCznWAiO7L54sbBhyyI23mcgVmMuz5YG0HTg2hhcPOG6EdD55c5JjgChCW3c= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: ee42da05-bd4e-4054-cf4b-08d63fcaa5ff X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Nov 2018 07:21:31.9612 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR05MB4107 Subject: Re: [dpdk-dev] [PATCH v4] mlx5: Support for rte_eth_rx_queue_count X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 07:21:34 -0000 Wednesday, October 31, 2018 11:01 AM, Tom Barbette: > Subject: RE: [PATCH v4] mlx5: Support for rte_eth_rx_queue_count >=20 > Hi Shahaf, >=20 > I don't see how rte_eth_rx_descriptor_status can actually give me the > number of packets in the RX queue? It will tell me the status of a packet= at a > given offset, right? It will tell you if in a given offset on the rxq you have a packet ready. I= think it will fit your needs, see below.=20 >=20 > About the goal: we have a full view of a network (switches and servers), = and > we want to know where the queueing is happening. So queues from > switches and servers are reported to the controller to deduce latency, > congestion, .... > On top of that, as CPU occupancy is somehow erratic with PMDs (ie we use > approximations), the queuing helps to make better scheduling decisions > about which servers could accept more load. So how about the below heuristic using rte_eth_rx_descriptor_status. Let's say you configure the rxq w/ N descriptors. Pick some threshold which= will represent "this queue on this server has enough work, no need to send= more", e.g. 3*N/4.=20 Monitor the rxq descriptor using rte_eth_rx_descriptor_status(port, rxq, 3*= N/4). If you get RTE_ETH_RX_DESC_AVAIL, then the TH is not yet reached and = the server can process more, otherwise if you get RTE_ETH_RX_DESC_DONE it m= eans you should stop scheduling packets for this one. You can also pick set of different TH to say queues is {not busy, partially= busy, really busy} to deduce about the latency. But for the latency it is= better to work w/ NIC host coherent clock + timestamps (like you implement= ed on a different patch). =20 >=20 > Considering the advance of Smart NICs, being able to monitor NICs as any > networking equipment in a network is of even more importance (if I still > need to make that point?). >=20 >=20 > Tom >=20 >=20 >=20 >=20 > ________________________________________ > De : Shahaf Shuler Envoy=E9 : dimanche 28 octobre > 2018 10:37 =C0 : Tom Barbette; dev@dpdk.org; Ferruh Yigit; Thomas Monjalo= n; > Andrew Rybchenko; olivier.matz@6wind.com Cc : Yongseok Koh Objet : RE: > [PATCH v4] mlx5: Support for rte_eth_rx_queue_count >=20 > Hi Tom, >=20 > Adding ethdev maintainers and Oliver as the author of the new API. >=20 >=20 > Saturday, October 27, 2018 6:11 PM, Tom Barbette: > > Subject: [PATCH v4] mlx5: Support for rte_eth_rx_queue_count > > >=20 > I have a more basic question. > The rte_eth_rx_queue_count is a very old API, more or less from the > beginning of DPDK. > We progressed from then and a newer API to check the descriptor status > was introduced @DPDK17.05 : rte_eth_rx_descriptor_status, see commit[1]. > There is also a plan to deprecate it once all drivers will support the ne= w API. >=20 > With the new API you can check the number of available descriptors on a > similar way. > So my question is, is the new API enough for your functionality? If not, = what > it is missing? I would prefer to improve the new one instead of starting = to > support the old one. >=20 >=20 > [1] > commit b1b700ce7d6fe0db9152f73e8e00394fc2756e60 > Author: Olivier Matz > Date: Wed Mar 29 10:36:28 2017 +0200 >=20 > ethdev: add descriptor status API >=20 > Introduce a new API to get the status of a descriptor. >=20 > For Rx, it is almost similar to rx_descriptor_done API, except it > differentiates "used" descriptors (which are hold by the driver and n= ot > returned to the hardware). >=20 > For Tx, it is a new API. >=20 > The descriptor_done() API, and probably the rx_queue_count() API coul= d > be replaced by this new API as soon as it is implemented on all PMDs. >=20 > Signed-off-by: Olivier Matz > Reviewed-by: Andrew Rybchenko >=20 >=20 > > This patch adds support for the rx_queue_count API in mlx5 driver > > > > Changes in v2: > > * Fixed styling issues > > * Fix missing return > > > > Changes in v3: > > * Fix styling comments and checks as per Yongseok Koh > > comments. Thanks ! > > > > Changes in v4: > > * Fix compiling issue because of a line that disappeared in v3 > > > > Signed-off-by: Tom Barbette > > --- > > drivers/net/mlx5/mlx5.c | 1 + > > drivers/net/mlx5/mlx5_rxtx.c | 78 > > ++++++++++++++++++++++++++++++++++++++------ > > drivers/net/mlx5/mlx5_rxtx.h | 1 + > > 3 files changed, 70 insertions(+), 10 deletions(-) > > > > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > > ec63bc6..6fccadd 100644 > > --- a/drivers/net/mlx5/mlx5.c > > +++ b/drivers/net/mlx5/mlx5.c > > @@ -375,6 +375,7 @@ const struct eth_dev_ops mlx5_dev_ops =3D { > > .filter_ctrl =3D mlx5_dev_filter_ctrl, > > .rx_descriptor_status =3D mlx5_rx_descriptor_status, > > .tx_descriptor_status =3D mlx5_tx_descriptor_status, > > + .rx_queue_count =3D mlx5_rx_queue_count, > > .rx_queue_intr_enable =3D mlx5_rx_intr_enable, > > .rx_queue_intr_disable =3D mlx5_rx_intr_disable, > > .is_removed =3D mlx5_is_removed, > > diff --git a/drivers/net/mlx5/mlx5_rxtx.c > > b/drivers/net/mlx5/mlx5_rxtx.c index 2d14f8a..2126205 100644 > > --- a/drivers/net/mlx5/mlx5_rxtx.c > > +++ b/drivers/net/mlx5/mlx5_rxtx.c > > @@ -417,20 +417,17 @@ mlx5_tx_descriptor_status(void *tx_queue, > > uint16_t offset) } > > > > /** > > - * DPDK callback to check the status of a rx descriptor. > > + * Internal function to compute the number of used descriptors in an > > + RX queue > > * > > - * @param rx_queue > > - * The rx queue. > > - * @param[in] offset > > - * The index of the descriptor in the ring. > > + * @param rxq > > + * The Rx queue. > > * > > * @return > > - * The status of the tx descriptor. > > + * The number of used rx descriptor. > > */ > > -int > > -mlx5_rx_descriptor_status(void *rx_queue, uint16_t offset) > > +static uint32_t > > +rx_queue_count(struct mlx5_rxq_data *rxq) > > { > > - struct mlx5_rxq_data *rxq =3D rx_queue; > > struct rxq_zip *zip =3D &rxq->zip; > > volatile struct mlx5_cqe *cqe; > > const unsigned int cqe_n =3D (1 << rxq->cqe_n); @@ -461,12 +458,7= 3 > > @@ mlx5_rx_descriptor_status(void *rx_queue, uint16_t offset) > > cqe =3D &(*rxq->cqes)[cq_ci & cqe_cnt]; > > } > > used =3D RTE_MIN(used, (1U << rxq->elts_n) - 1); > > - if (offset < used) > > + return used; > > +} > > + > > +/** > > + * DPDK callback to check the status of a rx descriptor. > > + * > > + * @param rx_queue > > + * The Rx queue. > > + * @param[in] offset > > + * The index of the descriptor in the ring. > > + * > > + * @return > > + * The status of the tx descriptor. > > + */ > > +int > > +mlx5_rx_descriptor_status(void *rx_queue, uint16_t offset) { > > + struct mlx5_rxq_data *rxq =3D rx_queue; > > + struct mlx5_rxq_ctrl *rxq_ctrl =3D > > + container_of(rxq, struct mlx5_rxq_ctrl, rxq); > > + struct rte_eth_dev *dev =3D ETH_DEV(rxq_ctrl->priv); > > + > > + if (dev->rx_pkt_burst !=3D mlx5_rx_burst) { > > + rte_errno =3D ENOTSUP; > > + return -rte_errno; > > + } > > + if (offset >=3D (1 << rxq->elts_n)) { > > + rte_errno =3D EINVAL; > > + return -rte_errno; > > + } > > + if (offset < rx_queue_count(rxq)) > > return RTE_ETH_RX_DESC_DONE; > > return RTE_ETH_RX_DESC_AVAIL; > > } > > > > /** > > + * DPDK callback to get the number of used descriptors in a RX queue > > + * > > + * @param dev > > + * Pointer to the device structure. > > + * > > + * @param rx_queue_id > > + * The Rx queue. > > + * > > + * @return > > + * The number of used rx descriptor. > > + * -EINVAL if the queue is invalid > > + */ > > +uint32_t > > +mlx5_rx_queue_count(struct rte_eth_dev *dev, uint16_t rx_queue_id) { > > + struct priv *priv =3D dev->data->dev_private; > > + struct mlx5_rxq_data *rxq; > > + > > + if (dev->rx_pkt_burst !=3D mlx5_rx_burst) { > > + rte_errno =3D ENOTSUP; > > + return -rte_errno; > > + } > > + rxq =3D (*priv->rxqs)[rx_queue_id]; > > + if (!rxq) { > > + rte_errno =3D EINVAL; > > + return -rte_errno; > > + } > > + return rx_queue_count(rxq); > > +} > > + > > +/** > > * DPDK callback for TX. > > * > > * @param dpdk_txq > > diff --git a/drivers/net/mlx5/mlx5_rxtx.h > > b/drivers/net/mlx5/mlx5_rxtx.h index 48ed2b2..c82059b 100644 > > --- a/drivers/net/mlx5/mlx5_rxtx.h > > +++ b/drivers/net/mlx5/mlx5_rxtx.h > > @@ -345,6 +345,7 @@ uint16_t removed_rx_burst(void *dpdk_rxq, struct > > rte_mbuf **pkts, > > uint16_t pkts_n); int > > mlx5_rx_descriptor_status(void *rx_queue, uint16_t offset); int > > mlx5_tx_descriptor_status(void *tx_queue, uint16_t offset); > > +uint32_t mlx5_rx_queue_count(struct rte_eth_dev *dev, uint16_t > > +rx_queue_id); > > > > /* Vectorized version of mlx5_rxtx.c */ int > > mlx5_check_raw_vec_tx_support(struct rte_eth_dev *dev); > > -- > > 2.7.4