From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-eopbgr150089.outbound.protection.outlook.com [40.107.15.89]) by dpdk.org (Postfix) with ESMTP id 333FE5398 for ; Tue, 9 Apr 2019 21:36:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iHQiLOpBs2b5TZLhbH/0YS+QuOrhxP0/2fK3FEp0xsA=; b=YauaXmEtAr1SUO/Sxogc9N+Bn30UbdCV4O+FF2Ib4kulO7+r4/lb6GYwEDXwOGtP28/f+M626HOCMol/AIxBhYpSO1eYBxLSzFUyTc5DEOvtu9Y2Tli1K4SzCxg/ifgD3q8gXexrNvRMqaECpeJKV2lAKPXkDw5OLI/DbL3gWC0= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4089.eurprd05.prod.outlook.com (52.134.68.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1771.15; Tue, 9 Apr 2019 19:36:51 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a%3]) with mapi id 15.20.1792.009; Tue, 9 Apr 2019 19:36:51 +0000 From: Yongseok Koh To: Shahaf Shuler CC: "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH v3 3/4] net/mlx5: remove device register remap Thread-Index: AQHU60/ANt0yGcae4E+B6NCX0MCzjKYxxsYAgAJ5wQA= Date: Tue, 9 Apr 2019 19:36:51 +0000 Message-ID: <0731AA80-EA07-4F93-B5DA-35FD442F0FDC@mellanox.com> References: <20190325193627.19726-1-yskoh@mellanox.com> <20190405013357.14503-1-yskoh@mellanox.com> <20190405013357.14503-4-yskoh@mellanox.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 76a4773f-1f92-488f-5a32-08d6bd22b70a x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600139)(711020)(4605104)(4618075)(2017052603328)(7193020); SRVR:DB3PR0502MB4089; x-ms-traffictypediagnostic: DB3PR0502MB4089: x-microsoft-antispam-prvs: x-forefront-prvs: 000227DA0C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(39860400002)(366004)(376002)(136003)(346002)(189003)(199004)(25786009)(6512007)(6636002)(81166006)(81156014)(105586002)(8676002)(36756003)(53936002)(82746002)(305945005)(14444005)(7736002)(106356001)(6246003)(5024004)(8936002)(68736007)(256004)(86362001)(99286004)(229853002)(33656002)(102836004)(2616005)(3846002)(6506007)(6116002)(14454004)(6862004)(446003)(486006)(53546011)(6486002)(476003)(316002)(11346002)(478600001)(186003)(83716004)(6436002)(37006003)(97736004)(2906002)(26005)(93886005)(66066001)(71190400001)(4326008)(71200400001)(5660300002)(76176011); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4089; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: CW5W9y6o66VTSYgiRQsNNDEQKCzVXWbboPIg4xHoGmX+L86KsBLm3R9hoKr9/yMyrL2CJRPsXY8243yGbYDJmpyZO6yeBtJY7rkOP6wbsFG7Cmor2l3JPeCAW+Y8liHg53m6E0lJmsbMkoMsxnR3JjhxCqcmIqrrAmiRFgIXR0vtE+77TDfeK8Inqoer8hiQVnRqRFz+J/Zm6+b01vGvZEOXkNGtgGdH6IOixBD+JoLwQFVN55IQhyZAXv8+UC0lgBfzmaivZm7/sVfzPlPvygew7kxbo/xquBtWZf+XsbcCFw8hI7Ccs6uS/i7uvpNC2//DidBE6qaX+lI1DR4X0D2KwImYzM3t26JEb59N118/cuQHjILTRXBl7XtzutsGaR2UfC77HD/ESMO1ZbYA/h034nKcCWC3+kWGZigOD4s= Content-Type: text/plain; charset="us-ascii" Content-ID: <67C58E6614B13E4C8E86FEE9E2E6EA22@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 76a4773f-1f92-488f-5a32-08d6bd22b70a X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Apr 2019 19:36:51.3668 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4089 Subject: Re: [dpdk-dev] [PATCH v3 3/4] net/mlx5: remove device register remap X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 19:36:53 -0000 > On Apr 7, 2019, at 10:48 PM, Shahaf Shuler wrote: >=20 > Hi Koh, >=20 > See small comments below. Same for mlx4 patch. >=20 >=20 > Friday, April 5, 2019 4:34 AM, Yongseok Koh: >> Subject: [dpdk-dev] [PATCH v3 3/4] net/mlx5: remove device register rema= p >>=20 >> UAR (User Access Region) register does not need to be remapped for >> primary process but it should be remapped only for secondary process. UA= R >> register table is in the process private structure in rte_eth_devices[], >> (struct mlx5_proc_priv *)rte_eth_devices[port_id].process_private >>=20 >> The actual UAR table follows the data structure and the table is used fo= r both >> Tx and Rx. >>=20 >> For Tx, BlueFlame in UAR is used to ring the doorbell. MLX5_TX_BFREG(txq= ) >> is defined to get a register for the txq. Processes access its own priva= te data >> to acquire the register from the UAR table. >>=20 >> For Rx, the doorbell in UAR is required in arming CQ event. However, it = is a >> known issue that the register isn't remapped for secondary process. >>=20 >> Signed-off-by: Yongseok Koh >> --- >> drivers/net/mlx5/mlx5.c | 198 +++++-----------------------------= ------ >> drivers/net/mlx5/mlx5.h | 15 ++- >> drivers/net/mlx5/mlx5_ethdev.c | 17 ++++ >> drivers/net/mlx5/mlx5_rxtx.h | 11 ++- >> drivers/net/mlx5/mlx5_trigger.c | 6 -- >> drivers/net/mlx5/mlx5_txq.c | 180 ++++++++++++++++++++++------------= - >> - >=20 > [...] >=20 >> /** >> @@ -1182,12 +1010,32 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, >> } >> DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); >> if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { >> + struct mlx5_proc_priv *ppriv; >> + size_t ppriv_size; >> + >> eth_dev =3D rte_eth_dev_attach_secondary(name); >> if (eth_dev =3D=3D NULL) { >> DRV_LOG(ERR, "can not attach rte ethdev"); >> rte_errno =3D ENOMEM; >> return NULL; >> } >> + priv =3D eth_dev->data->dev_private; >> + /* >> + * UAR register table follows the process private structure. >> + * BlueFlame registers for Tx queues come first and registers >> + * for Rx queues follows. >> + */ >> + ppriv_size =3D sizeof(struct mlx5_proc_priv) + >> + (priv->rxqs_n + priv->txqs_n) * sizeof(void *); >=20 > Why you add also the rxqs_n? why not only the txqs? I wanted to make a room for the registers for arming Rx CQ, which will be f= ixed soon. But, I think it would be better to avoid confusion in this patch. Wil= l remove. [...] >> + ppriv =3D rte_malloc_socket("mlx5_proc_priv", ppriv_size, >> + RTE_CACHE_LINE_SIZE, dev->device- >>> numa_node); >> + if (!ppriv) { >> + rte_errno =3D ENOMEM; >> + return -rte_errno; >> + } >> + ppriv->uar_table_sz =3D ppriv_size; >> + dev->process_private =3D ppriv; >> return 0; >> } >>=20 >> diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h >> index 7b58063ceb..5d49892429 100644 >> --- a/drivers/net/mlx5/mlx5_rxtx.h >> +++ b/drivers/net/mlx5/mlx5_rxtx.h >> @@ -201,8 +201,8 @@ struct mlx5_txq_data { >> volatile void *wqes; /* Work queue (use volatile to write into). */ >> volatile uint32_t *qp_db; /* Work queue doorbell. */ >> volatile uint32_t *cq_db; /* Completion queue doorbell. */ >> - volatile void *bf_reg; /* Blueflame register remapped. */ >> struct rte_mbuf *(*elts)[]; /* TX elements. */ >> + uint16_t port_id; /* Port ID of device. */ >> uint16_t idx; /* Queue index. */ >> struct mlx5_txq_stats stats; /* TX queue counters. */ #ifndef >> RTE_ARCH_64 @@ -231,9 +231,12 @@ struct mlx5_txq_ctrl { >> struct mlx5_txq_ibv *ibv; /* Verbs queue object. */ >> struct mlx5_priv *priv; /* Back pointer to private data. */ >> off_t uar_mmap_offset; /* UAR mmap offset for non-primary >> process. */ >> - volatile void *bf_reg_orig; /* Blueflame register from verbs. */ >> + void *bf_reg; /* BlueFlame register from Verbs. */ >=20 > I guess you keep this one in order to get the VA offset for the secondary= mapping, right? Because otherwise we can take the bf_reg from the UAR tabl= e on the process private. >=20 > If so, better to rename it to uar_page_offset (or other name you like) in= order to avoid fields duplication.=20 It doesn't have the offset (offset can be calculated when remapping). It is the original BF register address gotten from QP creation. >> }; >>=20 >> +#define MLX5_TX_BFREG(txq) \ >> + (MLX5_PROC_PRIV((txq)->port_id)->uar_table[(txq)->idx]) >> + >> /* mlx5_rxq.c */ >>=20 >> extern uint8_t rss_hash_default_key[]; >> @@ -301,7 +304,7 @@ uint64_t mlx5_get_rx_queue_offloads(struct >> rte_eth_dev *dev); int mlx5_tx_queue_setup(struct rte_eth_dev *dev, >> uint16_t idx, uint16_t desc, >> unsigned int socket, const struct rte_eth_txconf >> *conf); void mlx5_tx_queue_release(void *dpdk_txq); -int >> mlx5_tx_uar_remap(struct rte_eth_dev *dev, int fd); >> +int mlx5_tx_uar_init_secondary(struct rte_eth_dev *dev, int fd); >> struct mlx5_txq_ibv *mlx5_txq_ibv_new(struct rte_eth_dev *dev, uint16_t >> idx); struct mlx5_txq_ibv *mlx5_txq_ibv_get(struct rte_eth_dev *dev, >> uint16_t idx); int mlx5_txq_ibv_release(struct mlx5_txq_ibv *txq_ibv); = @@ >> -704,7 +707,7 @@ static __rte_always_inline void >> mlx5_tx_dbrec_cond_wmb(struct mlx5_txq_data *txq, volatile struct >> mlx5_wqe *wqe, >> int cond) >> { >> - uint64_t *dst =3D (uint64_t *)((uintptr_t)txq->bf_reg); >> + uint64_t *dst =3D MLX5_TX_BFREG(txq); >=20 > I guess no perf penalty due to this change right? I've confirmed no perf drop on x86 and ARM (BlueField). It might have been smaller than the jitter even if any. > Would you consider to prefetch the addr before the db logic just to be on= the safe side? As it is not a random/sequential access but accessing the same cacheline repeatedly, I wouldn't have a prefetch here. Sometimes, prefetch have a lit= tle cost. Will send out a new version just in case you agree and want to merge it. If you still want to change something, please comment on the new one. Thanks, Yongseok= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 3E052A0096 for ; Tue, 9 Apr 2019 21:36:56 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B64885424; Tue, 9 Apr 2019 21:36:54 +0200 (CEST) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-eopbgr150089.outbound.protection.outlook.com [40.107.15.89]) by dpdk.org (Postfix) with ESMTP id 333FE5398 for ; Tue, 9 Apr 2019 21:36:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iHQiLOpBs2b5TZLhbH/0YS+QuOrhxP0/2fK3FEp0xsA=; b=YauaXmEtAr1SUO/Sxogc9N+Bn30UbdCV4O+FF2Ib4kulO7+r4/lb6GYwEDXwOGtP28/f+M626HOCMol/AIxBhYpSO1eYBxLSzFUyTc5DEOvtu9Y2Tli1K4SzCxg/ifgD3q8gXexrNvRMqaECpeJKV2lAKPXkDw5OLI/DbL3gWC0= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4089.eurprd05.prod.outlook.com (52.134.68.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1771.15; Tue, 9 Apr 2019 19:36:51 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a%3]) with mapi id 15.20.1792.009; Tue, 9 Apr 2019 19:36:51 +0000 From: Yongseok Koh To: Shahaf Shuler CC: "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH v3 3/4] net/mlx5: remove device register remap Thread-Index: AQHU60/ANt0yGcae4E+B6NCX0MCzjKYxxsYAgAJ5wQA= Date: Tue, 9 Apr 2019 19:36:51 +0000 Message-ID: <0731AA80-EA07-4F93-B5DA-35FD442F0FDC@mellanox.com> References: <20190325193627.19726-1-yskoh@mellanox.com> <20190405013357.14503-1-yskoh@mellanox.com> <20190405013357.14503-4-yskoh@mellanox.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 76a4773f-1f92-488f-5a32-08d6bd22b70a x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600139)(711020)(4605104)(4618075)(2017052603328)(7193020); SRVR:DB3PR0502MB4089; x-ms-traffictypediagnostic: DB3PR0502MB4089: x-microsoft-antispam-prvs: x-forefront-prvs: 000227DA0C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(39860400002)(366004)(376002)(136003)(346002)(189003)(199004)(25786009)(6512007)(6636002)(81166006)(81156014)(105586002)(8676002)(36756003)(53936002)(82746002)(305945005)(14444005)(7736002)(106356001)(6246003)(5024004)(8936002)(68736007)(256004)(86362001)(99286004)(229853002)(33656002)(102836004)(2616005)(3846002)(6506007)(6116002)(14454004)(6862004)(446003)(486006)(53546011)(6486002)(476003)(316002)(11346002)(478600001)(186003)(83716004)(6436002)(37006003)(97736004)(2906002)(26005)(93886005)(66066001)(71190400001)(4326008)(71200400001)(5660300002)(76176011); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4089; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: CW5W9y6o66VTSYgiRQsNNDEQKCzVXWbboPIg4xHoGmX+L86KsBLm3R9hoKr9/yMyrL2CJRPsXY8243yGbYDJmpyZO6yeBtJY7rkOP6wbsFG7Cmor2l3JPeCAW+Y8liHg53m6E0lJmsbMkoMsxnR3JjhxCqcmIqrrAmiRFgIXR0vtE+77TDfeK8Inqoer8hiQVnRqRFz+J/Zm6+b01vGvZEOXkNGtgGdH6IOixBD+JoLwQFVN55IQhyZAXv8+UC0lgBfzmaivZm7/sVfzPlPvygew7kxbo/xquBtWZf+XsbcCFw8hI7Ccs6uS/i7uvpNC2//DidBE6qaX+lI1DR4X0D2KwImYzM3t26JEb59N118/cuQHjILTRXBl7XtzutsGaR2UfC77HD/ESMO1ZbYA/h034nKcCWC3+kWGZigOD4s= Content-Type: text/plain; charset="UTF-8" Content-ID: <67C58E6614B13E4C8E86FEE9E2E6EA22@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 76a4773f-1f92-488f-5a32-08d6bd22b70a X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Apr 2019 19:36:51.3668 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4089 Subject: Re: [dpdk-dev] [PATCH v3 3/4] net/mlx5: remove device register remap X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190409193651.mp3qOWpWPj4UQCtJnrgNLaL1mHkZcA4cgMvJVMbiEhc@z> > On Apr 7, 2019, at 10:48 PM, Shahaf Shuler wrote: >=20 > Hi Koh, >=20 > See small comments below. Same for mlx4 patch. >=20 >=20 > Friday, April 5, 2019 4:34 AM, Yongseok Koh: >> Subject: [dpdk-dev] [PATCH v3 3/4] net/mlx5: remove device register rema= p >>=20 >> UAR (User Access Region) register does not need to be remapped for >> primary process but it should be remapped only for secondary process. UA= R >> register table is in the process private structure in rte_eth_devices[], >> (struct mlx5_proc_priv *)rte_eth_devices[port_id].process_private >>=20 >> The actual UAR table follows the data structure and the table is used fo= r both >> Tx and Rx. >>=20 >> For Tx, BlueFlame in UAR is used to ring the doorbell. MLX5_TX_BFREG(txq= ) >> is defined to get a register for the txq. Processes access its own priva= te data >> to acquire the register from the UAR table. >>=20 >> For Rx, the doorbell in UAR is required in arming CQ event. However, it = is a >> known issue that the register isn't remapped for secondary process. >>=20 >> Signed-off-by: Yongseok Koh >> --- >> drivers/net/mlx5/mlx5.c | 198 +++++-----------------------------= ------ >> drivers/net/mlx5/mlx5.h | 15 ++- >> drivers/net/mlx5/mlx5_ethdev.c | 17 ++++ >> drivers/net/mlx5/mlx5_rxtx.h | 11 ++- >> drivers/net/mlx5/mlx5_trigger.c | 6 -- >> drivers/net/mlx5/mlx5_txq.c | 180 ++++++++++++++++++++++------------= - >> - >=20 > [...] >=20 >> /** >> @@ -1182,12 +1010,32 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, >> } >> DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); >> if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { >> + struct mlx5_proc_priv *ppriv; >> + size_t ppriv_size; >> + >> eth_dev =3D rte_eth_dev_attach_secondary(name); >> if (eth_dev =3D=3D NULL) { >> DRV_LOG(ERR, "can not attach rte ethdev"); >> rte_errno =3D ENOMEM; >> return NULL; >> } >> + priv =3D eth_dev->data->dev_private; >> + /* >> + * UAR register table follows the process private structure. >> + * BlueFlame registers for Tx queues come first and registers >> + * for Rx queues follows. >> + */ >> + ppriv_size =3D sizeof(struct mlx5_proc_priv) + >> + (priv->rxqs_n + priv->txqs_n) * sizeof(void *); >=20 > Why you add also the rxqs_n? why not only the txqs? I wanted to make a room for the registers for arming Rx CQ, which will be f= ixed soon. But, I think it would be better to avoid confusion in this patch. Wil= l remove. [...] >> + ppriv =3D rte_malloc_socket("mlx5_proc_priv", ppriv_size, >> + RTE_CACHE_LINE_SIZE, dev->device- >>> numa_node); >> + if (!ppriv) { >> + rte_errno =3D ENOMEM; >> + return -rte_errno; >> + } >> + ppriv->uar_table_sz =3D ppriv_size; >> + dev->process_private =3D ppriv; >> return 0; >> } >>=20 >> diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h >> index 7b58063ceb..5d49892429 100644 >> --- a/drivers/net/mlx5/mlx5_rxtx.h >> +++ b/drivers/net/mlx5/mlx5_rxtx.h >> @@ -201,8 +201,8 @@ struct mlx5_txq_data { >> volatile void *wqes; /* Work queue (use volatile to write into). */ >> volatile uint32_t *qp_db; /* Work queue doorbell. */ >> volatile uint32_t *cq_db; /* Completion queue doorbell. */ >> - volatile void *bf_reg; /* Blueflame register remapped. */ >> struct rte_mbuf *(*elts)[]; /* TX elements. */ >> + uint16_t port_id; /* Port ID of device. */ >> uint16_t idx; /* Queue index. */ >> struct mlx5_txq_stats stats; /* TX queue counters. */ #ifndef >> RTE_ARCH_64 @@ -231,9 +231,12 @@ struct mlx5_txq_ctrl { >> struct mlx5_txq_ibv *ibv; /* Verbs queue object. */ >> struct mlx5_priv *priv; /* Back pointer to private data. */ >> off_t uar_mmap_offset; /* UAR mmap offset for non-primary >> process. */ >> - volatile void *bf_reg_orig; /* Blueflame register from verbs. */ >> + void *bf_reg; /* BlueFlame register from Verbs. */ >=20 > I guess you keep this one in order to get the VA offset for the secondary= mapping, right? Because otherwise we can take the bf_reg from the UAR tabl= e on the process private. >=20 > If so, better to rename it to uar_page_offset (or other name you like) in= order to avoid fields duplication.=20 It doesn't have the offset (offset can be calculated when remapping). It is the original BF register address gotten from QP creation. >> }; >>=20 >> +#define MLX5_TX_BFREG(txq) \ >> + (MLX5_PROC_PRIV((txq)->port_id)->uar_table[(txq)->idx]) >> + >> /* mlx5_rxq.c */ >>=20 >> extern uint8_t rss_hash_default_key[]; >> @@ -301,7 +304,7 @@ uint64_t mlx5_get_rx_queue_offloads(struct >> rte_eth_dev *dev); int mlx5_tx_queue_setup(struct rte_eth_dev *dev, >> uint16_t idx, uint16_t desc, >> unsigned int socket, const struct rte_eth_txconf >> *conf); void mlx5_tx_queue_release(void *dpdk_txq); -int >> mlx5_tx_uar_remap(struct rte_eth_dev *dev, int fd); >> +int mlx5_tx_uar_init_secondary(struct rte_eth_dev *dev, int fd); >> struct mlx5_txq_ibv *mlx5_txq_ibv_new(struct rte_eth_dev *dev, uint16_t >> idx); struct mlx5_txq_ibv *mlx5_txq_ibv_get(struct rte_eth_dev *dev, >> uint16_t idx); int mlx5_txq_ibv_release(struct mlx5_txq_ibv *txq_ibv); = @@ >> -704,7 +707,7 @@ static __rte_always_inline void >> mlx5_tx_dbrec_cond_wmb(struct mlx5_txq_data *txq, volatile struct >> mlx5_wqe *wqe, >> int cond) >> { >> - uint64_t *dst =3D (uint64_t *)((uintptr_t)txq->bf_reg); >> + uint64_t *dst =3D MLX5_TX_BFREG(txq); >=20 > I guess no perf penalty due to this change right? I've confirmed no perf drop on x86 and ARM (BlueField). It might have been smaller than the jitter even if any. > Would you consider to prefetch the addr before the db logic just to be on= the safe side? As it is not a random/sequential access but accessing the same cacheline repeatedly, I wouldn't have a prefetch here. Sometimes, prefetch have a lit= tle cost. Will send out a new version just in case you agree and want to merge it. If you still want to change something, please comment on the new one. Thanks, Yongseok=