From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2CD84A0548; Tue, 20 Apr 2021 13:16:59 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A836041749; Tue, 20 Apr 2021 13:16:58 +0200 (CEST) Received: from mail-ed1-f47.google.com (mail-ed1-f47.google.com [209.85.208.47]) by mails.dpdk.org (Postfix) with ESMTP id AC43D41735 for ; Tue, 20 Apr 2021 13:16:56 +0200 (CEST) Received: by mail-ed1-f47.google.com with SMTP id h10so44538193edt.13 for ; Tue, 20 Apr 2021 04:16:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=WUH2LCdTRPGs22TRqLxVDe9pKUFhmXGf4LWVMyMoGPA=; b=YW6LRFjTo3j9FO+r9yO4H6EyyB9nCXCixD7/af9b4UrKBNjysl90xVifid7AwGz45y X6hFxNlE+QPJ8lifopizy7KIq4k32oHpaicCQYjGe/Ex5W23+dd2jCcF3VTQqF8yCEfW QzjUHt83uMzvh0xOyIaVEnNt0YpHwKMcGdkVxg4t0pICjk7P33KNEYwxIR4UpQXrrivq 1yBtCdUU83sVjQL//xy5Xw6PosKaEFsFOZWyIQ5fSYGKChwASe8bhxGcxVwfVKuBefBV QZ0httlHm6eR/1Dq2HKGa51JNBzbOmiEN/WgNZkGTTAZKRcDU7fSFh+ShbOYM2ZKt5Bc ds7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=WUH2LCdTRPGs22TRqLxVDe9pKUFhmXGf4LWVMyMoGPA=; b=CtRWIB3PFLnK7Xnd7wMEr+B1CTp+TSz7uRwARGJmqC4WgOywIEuAyMelk6k+NQn7D8 RIbLxM6tsl7OHX1s55lmVTOvZB9gK14X36Q9lGlxouwJ5Tmz/Qk1ZMwBRtM7OA0EgDbn 0XzJzg7rb383lZjrKJgSdloXh5kKEG/dWvnypPB8vh3lPii79qYVV+pdnf7BtQtkhvy3 wafo46IS8PemNTJ9epui748bC37aEu7K0tTDwgm3U80UWRfmtjtc2gR78RvJVp8F108X 9pcI5jsJxsRJvzjrMDaiE/XX+bgsGY1whEGxJBRWdbNdilkA80Ve/o3W9X4RcGrPyFpc zYMg== X-Gm-Message-State: AOAM531uBWQmEktJA3l6ueIEuOLG3Od0YeK/hthGAvTwHppf+5WSAXRm JlDgR6xpleg5TumURqDY5GrXQRK6ceigTH77ARhVLi/U7C8/kA== X-Google-Smtp-Source: ABdhPJxVWSAE8w3vU4wSBDei67FuDetoqjTE439wwzwLz1o22WUERZd9n7UecsGkbZcyzqJO8sUQu2byUTSwJQm7FDQ= X-Received: by 2002:a05:6402:51d4:: with SMTP id r20mr31706520edd.112.1618917415840; Tue, 20 Apr 2021 04:16:55 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Rajesh Kumar Date: Tue, 20 Apr 2021 16:46:44 +0530 Message-ID: To: dev@dpdk.org, rajesh.kalidass@gigamon.com, suanmingm@nvidia.com, rakesh.jagota@gigamon.com, tanmay.kishore@gigamon.com, hung.nguyen@gigamon.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [dpdk-dev] net/mlx5: no TX in multi-process setup (testpmd working) X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" I backported the fix from suanmingm@nvidia.com(related to secondary process) and it worked !!! >From 1676903aea413fe8be4138b285633e01332efa17 Mon Sep 17 00:00:00 2001 From: RajeshKumar Kalidass Date: Tue, 20 Apr 2021 02:56:32 -0700 Subject: [PATCH] VM-16160 mlx5: secondary not able to transmit out pkt Change-Id: I647ba4f4d2534c2c97b5e23ce8a11a20eac207a3 --- diff --git a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.c b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.c index baffa75..0bfaddb 100644 --- a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.c +++ b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.c @@ -1183,7 +1183,7 @@ rte_errno =3D ENOMEM; return -rte_errno; } - ppriv->uar_table_sz =3D ppriv_size; + ppriv->uar_table_sz =3D priv->txqs_n; dev->process_private =3D ppriv; return 0; } @@ -1194,7 +1194,7 @@ * @param dev * Pointer to Ethernet device structure. */ -static void +void mlx5_proc_priv_uninit(struct rte_eth_dev *dev) { if (!dev->process_private) @@ -2036,24 +2036,6 @@ } return 0; } -int -mlx5_uar_table_init (struct rte_eth_dev *eth_dev) -{ - int err =3D 0; - int fd; - /* Receive command fd from primary process. */ - fd =3D mlx5_mp_req_verbs_cmd_fd(eth_dev); - if (fd < 0) { - return (rte_errno); - } - /* Remap UAR for Tx queues. */ - err =3D mlx5_tx_uar_init_secondary(eth_dev, fd); - if (err) { - err =3D rte_errno; - } - - return err; -} /** * Spawn an Ethernet device from Verbs information. * diff --git a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.h b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.h index 0c3a90e..5230ad6 100644 --- a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.h +++ b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5.h @@ -793,6 +793,7 @@ int mlx5_getenv_int(const char *); int mlx5_proc_priv_init(struct rte_eth_dev *dev); +void mlx5_proc_priv_uninit(struct rte_eth_dev *dev); int64_t mlx5_get_dbr(struct rte_eth_dev *dev, struct mlx5_devx_dbr_page **dbr_page); int32_t mlx5_release_dbr(struct rte_eth_dev *dev, uint32_t umem_id, diff --git a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_mp.c b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_mp.c index b65b019..3e8a030 100644 --- a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_mp.c +++ b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_mp.c @@ -119,6 +119,8 @@ const struct mlx5_mp_param *param =3D (const struct mlx5_mp_param *)mp_msg->param; struct rte_eth_dev *dev; + struct mlx5_proc_priv *ppriv; + struct mlx5_priv *priv; int ret; assert(rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY); @@ -128,12 +130,28 @@ return -rte_errno; } dev =3D &rte_eth_devices[param->port_id]; + priv =3D dev->data->dev_private; switch (param->type) { case MLX5_MP_REQ_START_RXTX: DRV_LOG(INFO, "port %u starting datapath", dev->data->port_id); rte_mb(); dev->rx_pkt_burst =3D mlx5_select_rx_function(dev); dev->tx_pkt_burst =3D mlx5_select_tx_function(dev); + ppriv =3D (struct mlx5_proc_priv *)dev->process_private; + /* If Tx queue number changes, re-initialize UAR. */ + if (ppriv->uar_table_sz !=3D priv->txqs_n) { + mlx5_tx_uar_uninit_secondary(dev); + mlx5_proc_priv_uninit(dev); + ret =3D mlx5_proc_priv_init(dev); + if (ret) + return -rte_errno; + ret =3D mlx5_tx_uar_init_secondary(dev, mp_msg->fds[0]); + if (ret) { + mlx5_proc_priv_uninit(dev); + return -rte_errno; + } + } + mp_init_msg(dev, &mp_res, param->type); res->result =3D 0; ret =3D rte_mp_reply(&mp_res, peer); @@ -175,6 +193,7 @@ struct timespec ts =3D {.tv_sec =3D MLX5_MP_REQ_TIMEOUT_SEC, .tv_nsec =3D= 0}; int ret; int i; + struct mlx5_priv *priv; assert(rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY); if (!mlx5_shared_data->secondary_cnt) @@ -184,7 +203,12 @@ dev->data->port_id, type); return; } + priv =3D dev->data->dev_private; mp_init_msg(dev, &mp_req, type); + if (type =3D=3D MLX5_MP_REQ_START_RXTX) { + mp_req.num_fds =3D 1; + mp_req.fds[0] =3D ((struct ibv_context *)priv->sh->ctx)->cmd_fd; + } ret =3D rte_mp_request_sync(&mp_req, &mp_rep, &ts); if (ret) { if (rte_errno !=3D ENOTSUP) diff --git a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_rxtx.h b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_rxtx.h index e927343..d8b3220 100644 --- a/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_rxtx.h +++ b/dpdk/dpdk-19.11/drivers/net/mlx5/mlx5_rxtx.h @@ -424,6 +424,7 @@ const struct rte_eth_hairpin_conf *hairpin_conf); void mlx5_tx_queue_release(void *dpdk_txq); int mlx5_tx_uar_init_secondary(struct rte_eth_dev *dev, int fd); +void mlx5_tx_uar_uninit_secondary(struct rte_eth_dev *dev); struct mlx5_txq_obj *mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx, enum mlx5_txq_obj_type type); struct mlx5_txq_obj *mlx5_txq_obj_get(struct rte_eth_dev *dev, uint16_t idx); On Mon, Apr 19, 2021 at 7:05 PM Rajesh Kumar wrote= : > Hi, > > > > DPDK: 19.11 > > OS: Ubuntu 18.04 (Kernel: 5.4.0-1043-azure) > > Iface: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual > Function] > > > > We are bringing-up our dpdk based app on azure cloud, its multi-process > setup (primary does dev_configure & dev_start ) =E2=80=93 however no pack= et are > getting transmitted out (Tx-packet increases upto number of descriptor an= d > then all further packets are txDropped) > > > > "stats": [ > > { > > "name": "rep1", > > "txPkts": 1024, <<<<<<----------------------- it > increases upto number of tx-descriptors > > "rxPkts": 5408, > > "txBytes": 65536, > > "rxBytes": 346112, > > "txDropped": 4384, <<<<<<--------------------- All further packets > are txDropped > > "rxDropped": 96, > > "txErrors": 0, > > "rxErrors": 0 > > } > > ] > > > > However mlx4 driver is working perfectly fine on multi-process setup. > Also testpmd working fine with mlx5. I guess problem is when we try to ru= n > in multi-process setup ? > > > > > > Thanks, > > *-Rajesh* >