From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C26ABA0547; Wed, 21 Apr 2021 19:23:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AA39141B9C; Wed, 21 Apr 2021 19:23:29 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2097.outbound.protection.outlook.com [40.107.220.97]) by mails.dpdk.org (Postfix) with ESMTP id 3FAFD41735 for ; Tue, 20 Apr 2021 16:51:57 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iNZDnA0a/DDl8lCMz3/+lu8GJbMTilcLytxXCZzk11iK4Qhs4ICFaecAoL5+WMwDNfpaLXO17LRGo8k31EfgzgTn+ATId4ePI2jZ24V9MFKpAoiHOorhPwzrGqFmzlZOZIJHROEZbCLbQMISGRD8VC3QFlHzo+zphAS3XnJ7L5bSgP6QMLeRN/bVeYM1Rf+atxkbklNrRY/PBKlkdif35uljVnTF+wooNc9pndg/ycIqEtuU1B/5wssI2dLST9wTF/HwwSbA8u4Np2otYt6Iuzfo5cAU8eLKKO2NDItDaY018mLAYf8HWmkGFzRMnnmKR62sC0Nl8fIg7TTJ89beBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=opEQi1A80OWvXEwkXxh3DZu/yYH/7K92wAb8bIhgTvY=; b=knYG9XbB++rBt+4EAe4ArMion6fxydtDai7x/YNi5QCV9da2HNyuNAwES1oEBthwdeRlzJ4OIN3ggIdHs6Lo+XsuNFTT0EXq3jfptVMw8j/5H1wbhaUFwVaba0FBdQZkMHlRKOFDLDozideIATeutHhwDXC0QK2+b/MGRtgvs2YAFdhp4PBAeyGNDwXtcp+JCvM+yI4O/EeItP48tivwW8LHUWpQ7loFM0zAkoM+rYWVldAooRt3ofFLx/8kgRuRgcXOu6PitHGem/0+I3J9FZDRG+m0+oO8sdMr6TAmogT2n0ulNRajC5J2+rvtbZo8bF7qeKpawfqlgLsTwiNBNA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=gigamon.com; dmarc=pass action=none header.from=gigamon.com; dkim=pass header.d=gigamon.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gigamon.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=opEQi1A80OWvXEwkXxh3DZu/yYH/7K92wAb8bIhgTvY=; b=mqSFfO2vbiVaKuEK46LySKPUON5uCs7IYjwjZmPQNgXdiCYq5Q5/EVUcqoeU4XZB3UKThEd7nriu15Ox3Il0/HDx4b1+Qvp9amZ6QScjynYmPqGUqcXkxvUSMmCP9i9S9qtl99+B1F9aVHNxflbt5iqXYxnQBlh8pW9WzVBiGmc= Received: from BYAPR12MB2822.namprd12.prod.outlook.com (2603:10b6:a03:9a::17) by BYAPR12MB3063.namprd12.prod.outlook.com (2603:10b6:a03:d6::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4042.16; Tue, 20 Apr 2021 14:51:55 +0000 Received: from BYAPR12MB2822.namprd12.prod.outlook.com ([fe80::25fa:8aba:7843:7e7]) by BYAPR12MB2822.namprd12.prod.outlook.com ([fe80::25fa:8aba:7843:7e7%3]) with mapi id 15.20.4065.020; Tue, 20 Apr 2021 14:51:55 +0000 From: Tanmay Kishore To: Rajesh Kumar , "dev@dpdk.org" , RajeshKumar Kalidass , "suanmingm@nvidia.com" , Rakesh Jagota , Hung Nguyen Thread-Topic: net/mlx5: no TX in multi-process setup (testpmd working) Thread-Index: AQHXNdauiQDAvmjU5Uehn4Z2S3kTPaq9fZaT Date: Tue, 20 Apr 2021 14:51:55 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=gigamon.com; x-originating-ip: [147.92.89.249] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3323c400-ccab-49d5-1057-08d9040bd77e x-ms-traffictypediagnostic: BYAPR12MB3063: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7219; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: FaAb63OhFoYA/0yINiZrGbmkEEkfiKQVg+kHPxuz5KTac1pjwM2y8NNEyGXb2HE92s7q8JlcjU0mYXel6wfDNjbY+tzSiT7r9dSv3Xot6xUWICKDN5/Uo2YFjIBhxYXw20h/Xfvy4brIA/oUO0J8ye9GMlbgBHbgWKvwH+nxeX2XGc06Fv3UZ7tXIOq/mD1fcRRzHvGclhS4wJmJ3lXTHH/VWiKk8iwUg8uVDT/n/ygCe6fPLd3LSVR9Ynm7S1OtqXChKCyCFP5LHTw1Hi1tSGPr8Ax96TZm0JiiU9Qr29r5kOnNCBAa74TKJNa35iZ9BGl/+Up7MfqG0mLBjKCTCMLP9oYTX7W/D+5aoeKw0zuBaVUTTRBhPt8NmcmIMWvvBKFV9x/zF6u7guFolmYTBKYwpl4U09VA43gj24S8ulOZj/xYeDFaZYzCwJq1nLkFW6n5fHkWnluexaIZjDePFErSNrPuxxU4KTq+nJEsDMUIve2yf38JOzaezJ6tUrP3+aOTMQq77GQp+62JfXclZViFB9ftHcRb4PhMedzfFn23tZuMm4OhOZ3+VvHotwSknmjR6GTySUTd+uW5m4QHkxT4jOksjiNEuOTg6xae/pU= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR12MB2822.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(39850400004)(136003)(346002)(396003)(366004)(376002)(33656002)(66446008)(5660300002)(64756008)(6636002)(186003)(53546011)(9686003)(86362001)(83380400001)(55016002)(316002)(52536014)(66556008)(66476007)(478600001)(76116006)(2906002)(38100700002)(66946007)(71200400001)(44832011)(8676002)(6506007)(26005)(122000001)(110136005)(8936002)(7696005); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: =?Windows-1252?Q?IGAB9AZjRtn5dQwI2ceSZLp8IyDLCT4vs62Y+V1J6vJ0s6Tc6+lvH5TS?= =?Windows-1252?Q?+oFQHGiIX7KXRTFCwVVEi/0+5yJKXGi+E29fqey+xhDbv0eKvSW19Brw?= =?Windows-1252?Q?h45RPO9EnvWfpX88BUlsOkM7lYRfOCtPqcIl2Ly7wn/4fvazhCR6ZGxl?= =?Windows-1252?Q?NEf2bc9iRVM27cDSRdIpmQyURJ+bHWO4/eMD46dWX8HB6bV2OU77tgjp?= =?Windows-1252?Q?QTy38Y7A76WutDC67VqjENu/J5U0r7AhxTTk5tSeIko2vqtn0OvTJq39?= =?Windows-1252?Q?NstlBpslbFoirHl8oAJqJMaBrLVHt3xcFueoy61CmXofUkp/m/Kff8qj?= =?Windows-1252?Q?Z8H7+cFos0g1qgqBY2OC42ZgNy5S36G6vQUBV0QCInE7Y1BZGkeONlT+?= =?Windows-1252?Q?WxyQbnrhQ4wLnKdTx4TKjl5iWV9QrRJ1q5WHl01RpmEWKBDnYM4XoXyC?= =?Windows-1252?Q?ROS2f28g4bccj2MHWhgF/lNAJMZIgqi47iY2YmBY6c3Oct8RgWrwnOlr?= =?Windows-1252?Q?SupI0rmv+0IQAes+6Th5l4W6GXgZCjYgN3eyQZjbOzJmkLTJ46IgxOP+?= =?Windows-1252?Q?Jx+bsIk56cv4CP2XB9FtjVKvmdZJPnzeDvyhAIQ9ZlmUfXywoCv3+w7k?= =?Windows-1252?Q?zurDN2rvIl38UB7mPy9zg8EpM9lQfB8PC2s3gseUlPkPQJvVbO+Wn+xr?= =?Windows-1252?Q?vv0GupwRW+G5jV3NZWv6h7HHXp5SBPM5k2SFPIUlknectdCKSMkhrvNJ?= =?Windows-1252?Q?3tWZ2yeYdyajxsTEKH8Z/rKwfcR2F78pkdNwDzDK1Tggm1O13bCQ6U8Q?= =?Windows-1252?Q?QYtx5AyRZzZXLia2zkME3yAbHJLFuHewDWlt8iC6ajqKRLz636Ckm6Jt?= =?Windows-1252?Q?2SJy5LJcKX43YW/VlQ0nxvO7CCX1hHwfQTz81xTGuyvcJYLqgJ23R29y?= =?Windows-1252?Q?afFoNk01pD0ldImrD2Hw9/gHhwv6RTZfYYO68nnysxlMs34AB97w9+RC?= =?Windows-1252?Q?VBDaz9yDGTrPWHpxQ2WvK+Jsxm+oD6uMeHk2gwjIvOGiia62b/7sgoso?= =?Windows-1252?Q?3Y9KZO0JZKxCWmZKNn6nNKZ0V6HzK/U/racXd6rnHmk8nExoPAmJ5cqA?= =?Windows-1252?Q?r28pen/qm11mQqZrq036k3fTCFPAOOfbrAAHJO8MrvEBtML7WnJDC3Fb?= =?Windows-1252?Q?1L/Ups7bp0y9+wKA3BvvDYUGm9foLq2dpkVxVCq1R2yCSXDtlOXr3+zd?= =?Windows-1252?Q?qnn2jWY/5YGggbTd6FEjIhpdYihVxqa0CW98hxZw6iwiXdzzaSlJV43U?= =?Windows-1252?Q?SyQ+G0eNKE921HCXhnB15otrazZ9GpeL7H5Li8lSgTUMYImsmhdBx1iU?= =?Windows-1252?Q?7EZMxYkXXpCKvs6ZAVTNG5nMppkmMl1D1/w=3D?= MIME-Version: 1.0 X-OriginatorOrg: gigamon.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB2822.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3323c400-ccab-49d5-1057-08d9040bd77e X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Apr 2021 14:51:55.3721 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 0927b619-f6e4-4461-bcda-333194268117 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: cRzrOn1CRTjylQeQhiXpT4670GEOMXAnozpTNywjBZG03nF6XOq+hALMHp9E1kXdqNUCo6DdVDnaRkm14GmHiQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB3063 X-Mailman-Approved-At: Wed, 21 Apr 2021 19:23:28 +0200 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [dpdk-dev] net/mlx5: no TX in multi-process setup (testpmd working) X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Nice!!! Tanmay ________________________________ From: Rajesh Kumar Sent: Tuesday, April 20, 2021 4:16 AM To: dev@dpdk.org ; RajeshKumar Kalidass ; suanmingm@nvidia.com ; Rakesh Jagota ; Tanmay Kishore ; Hung Ng= uyen Subject: Re: net/mlx5: no TX in multi-process setup (testpmd working) WARNING: This email originated from outside of Gigamon. Use caution with an= y links or attachments. I backported the fix from suanmingm@nvidia.com= (related to secondary process) and it worked !!! >From 1676903aea413fe8be4138b285633e01332efa17 Mon Sep 17 00:00:00 2001 From: RajeshKumar Kalidass > Date: Tue, 20 Apr 2021 02:56:32 -0700 Subject: [PATCH] VM-16160 mlx5: secondary not able to transmit out pkt Change-Id: I647ba4f4d2534c2c97b5e23ce8a11a20eac207a3 --- diff --git a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.c b/dpdk/dpdk-19.11/driv= ers/net/mlx5/mlx5.c index baffa75..0bfaddb 100644 --- a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.c +++ b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.c @@ -1183,7 +1183,7 @@ rte_errno =3D ENOMEM; return -rte_errno; } - ppriv->uar_table_sz =3D ppriv_size; + ppriv->uar_table_sz =3D priv->txqs_n; dev->process_private =3D ppriv; return 0; } @@ -1194,7 +1194,7 @@ * @param dev * Pointer to Ethernet device structure. */ -static void +void mlx5_proc_priv_uninit(struct rte_eth_dev *dev) { if (!dev->process_private) @@ -2036,24 +2036,6 @@ } return 0; } -int -mlx5_uar_table_init (struct rte_eth_dev *eth_dev) -{ - int err =3D 0; - int fd; - /* Receive command fd from primary process. */ - fd =3D mlx5_mp_req_verbs_cmd_fd(eth_dev); - if (fd < 0) { - return (rte_errno); - } - /* Remap UAR for Tx queues. */ - err =3D mlx5_tx_uar_init_secondary(eth_dev, fd); - if (err) { - err =3D rte_errno; - } - - return err; -} /** * Spawn an Ethernet device from Verbs information. * diff --git a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.h b/dpdk/dpdk-19.11/driv= ers/net/mlx5/mlx5.h index 0c3a90e..5230ad6 100644 --- a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.h +++ b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.h @@ -793,6 +793,7 @@ int mlx5_getenv_int(const char *); int mlx5_proc_priv_init(struct rte_eth_dev *dev); +void mlx5_proc_priv_uninit(struct rte_eth_dev *dev); int64_t mlx5_get_dbr(struct rte_eth_dev *dev, struct mlx5_devx_dbr_page **dbr_page); int32_t mlx5_release_dbr(struct rte_eth_dev *dev, uint32_t umem_id, diff --git a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_mp.c b/dpdk/dpdk-19.11/d= rivers/net/mlx5/mlx5_mp.c index b65b019..3e8a030 100644 --- a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_mp.c +++ b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_mp.c @@ -119,6 +119,8 @@ const struct mlx5_mp_param *param =3D (const struct mlx5_mp_param *)mp_msg->param; struct rte_eth_dev *dev; + struct mlx5_proc_priv *ppriv; + struct mlx5_priv *priv; int ret; assert(rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY); @@ -128,12 +130,28 @@ return -rte_errno; } dev =3D &rte_eth_devices[param->port_id]; + priv =3D dev->data->dev_private; switch (param->type) { case MLX5_MP_REQ_START_RXTX: DRV_LOG(INFO, "port %u starting datapath", dev->data->port_id); rte_mb(); dev->rx_pkt_burst =3D mlx5_select_rx_function(dev); dev->tx_pkt_burst =3D mlx5_select_tx_function(dev); + ppriv =3D (struct mlx5_proc_priv *)dev->process_private; + /* If Tx queue number changes, re-initialize UAR. */ + if (ppriv->uar_table_sz !=3D priv->txqs_n) { + mlx5_tx_uar_uninit_secondary(dev); + mlx5_proc_priv_uninit(dev); + ret =3D mlx5_proc_priv_init(dev); + if (ret) + return -rte_errno; + ret =3D mlx5_tx_uar_init_secondary(dev, mp_msg->fds[0]); + if (ret) { + mlx5_proc_priv_uninit(dev); + return -rte_errno; + } + } + mp_init_msg(dev, &mp_res, param->type); res->result =3D 0; ret =3D rte_mp_reply(&mp_res, peer); @@ -175,6 +193,7 @@ struct timespec ts =3D {.tv_sec =3D MLX5_MP_REQ_TIMEOUT_SEC, .tv_nsec =3D= 0}; int ret; int i; + struct mlx5_priv *priv; assert(rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY); if (!mlx5_shared_data->secondary_cnt) @@ -184,7 +203,12 @@ dev->data->port_id, type); return; } + priv =3D dev->data->dev_private; mp_init_msg(dev, &mp_req, type); + if (type =3D=3D MLX5_MP_REQ_START_RXTX) { + mp_req.num_fds =3D 1; + mp_req.fds[0] =3D ((struct ibv_context *)priv->sh->ctx)->cmd_fd; + } ret =3D rte_mp_request_sync(&mp_req, &mp_rep, &ts); if (ret) { if (rte_errno !=3D ENOTSUP) diff --git a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_rxtx.h b/dpdk/dpdk-19.11= /drivers/net/mlx5/mlx5_rxtx.h index e927343..d8b3220 100644 --- a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_rxtx.h +++ b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_rxtx.h @@ -424,6 +424,7 @@ const struct rte_eth_hairpin_conf *hairpin_conf); void mlx5_tx_queue_release(void *dpdk_txq); int mlx5_tx_uar_init_secondary(struct rte_eth_dev *dev, int fd); +void mlx5_tx_uar_uninit_secondary(struct rte_eth_dev *dev); struct mlx5_txq_obj *mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t id= x, enum mlx5_txq_obj_type type); struct mlx5_txq_obj *mlx5_txq_obj_get(struct rte_eth_dev *dev, uint16_t id= x); On Mon, Apr 19, 2021 at 7:05 PM Rajesh Kumar > wrote: Hi, DPDK: 19.11 OS: Ubuntu 18.04 (Kernel: 5.4.0-1043-azure) Iface: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function= ] We are bringing-up our dpdk based app on azure cloud, its multi-process set= up (primary does dev_configure & dev_start ) =96 however no packet are gett= ing transmitted out (Tx-packet increases upto number of descriptor and then= all further packets are txDropped) "stats": [ { "name": "rep1", "txPkts": 1024, <<<<<<----------------------- it increases = upto number of tx-descriptors "rxPkts": 5408, "txBytes": 65536, "rxBytes": 346112, "txDropped": 4384, <<<<<<--------------------- All further packets a= re txDropped "rxDropped": 96, "txErrors": 0, "rxErrors": 0 } ] However mlx4 driver is working perfectly fine on multi-process setup. Also = testpmd working fine with mlx5. I guess problem is when we try to run in mu= lti-process setup ? Thanks, -Rajesh This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. NOTE that all incoming emails sen= t to Gigamon email accounts will be archived and may be scanned by us and/o= r by external service providers to detect and prevent threats to our system= s, investigate illegal or inappropriate behavior, and/or eliminate unsolici= ted promotional emails (=93spam=94).