From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <yskoh@mellanox.com> Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70054.outbound.protection.outlook.com [40.107.7.54]) by dpdk.org (Postfix) with ESMTP id 0F37B37A2 for <dev@dpdk.org>; Thu, 28 Mar 2019 20:01:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gknYw0YKpt3DEQOAe8slqiY6cfMaRApRIweIVLaBcGU=; b=jaqNK6OI5pinQSNzrAORqwjM27KH54GLHxg8Y8Dsof25RmDQCwweBH70J7w4MJyacCzu9lepMWFrciUUsrGQsRfnteoeCXOcvwnXQyTCEp/VOqr6dJLPL/6xo4PzPzDdchQtiM4syDPEqAUMD6npRWtJEXcPt7dl2nU+VgseogI= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4057.eurprd05.prod.outlook.com (52.134.67.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1750.17; Thu, 28 Mar 2019 19:01:46 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a%3]) with mapi id 15.20.1750.014; Thu, 28 Mar 2019 19:01:46 +0000 From: Yongseok Koh <yskoh@mellanox.com> To: Shahaf Shuler <shahafs@mellanox.com> CC: "dev@dpdk.org" <dev@dpdk.org> Thread-Topic: [PATCH v2 3/3] net/mlx4: add secondary process support Thread-Index: AQHU4z+VDM+Spv/b9kKBel6MIVGd9qYeTykAgAMbt4A= Date: Thu, 28 Mar 2019 19:01:46 +0000 Message-ID: <41E63A87-1829-4447-AC43-09450B3AF3A7@mellanox.com> References: <20190307073905.18615-1-yskoh@mellanox.com> <20190325191801.20841-1-yskoh@mellanox.com> <20190325191801.20841-4-yskoh@mellanox.com> <AM0PR0502MB3795C2E69DB3E4A2F221461FC35F0@AM0PR0502MB3795.eurprd05.prod.outlook.com> In-Reply-To: <AM0PR0502MB3795C2E69DB3E4A2F221461FC35F0@AM0PR0502MB3795.eurprd05.prod.outlook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [69.181.245.183] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d68f6b04-0c66-4109-81ce-08d6b3afd351 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600127)(711020)(4605104)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB4057; x-ms-traffictypediagnostic: DB3PR0502MB4057: x-microsoft-antispam-prvs: <DB3PR0502MB40573C6A2E004A65143CC230C3590@DB3PR0502MB4057.eurprd05.prod.outlook.com> x-forefront-prvs: 0990C54589 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(366004)(396003)(39860400002)(376002)(136003)(189003)(199004)(14444005)(6512007)(6636002)(97736004)(186003)(53936002)(2906002)(99286004)(25786009)(478600001)(6436002)(2616005)(11346002)(6486002)(305945005)(446003)(7736002)(86362001)(6862004)(486006)(5660300002)(8936002)(106356001)(33656002)(105586002)(6246003)(36756003)(66066001)(68736007)(71200400001)(71190400001)(83716004)(53546011)(82746002)(4326008)(6506007)(26005)(476003)(102836004)(316002)(37006003)(93886005)(76176011)(14454004)(81156014)(8676002)(6116002)(3846002)(81166006)(229853002)(256004); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4057; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: xTWCGG2Jz4I/A3BJGL0pvU2jKtAiYeK0CT0lxvyItQ4Kb4+HrSdc26ANd7xjy4hCNJsT6XN+UvawpMDPtX4LTRuBmb0bv50/TlvYv69NG2TDtVCJTaHwBni9K296GzsXEY7SgqHVDsr4YFN4cJvEWQjE1Wpgo710mx0qfqD3I3LbKu2Wtha54mJnpxcIstHW6yO9CqvsgHsu21u7GH6NdQLp7jIUP2cGVjMwmtHqbAKWNGZS2gY5Z4nB+GLcexG7YAFX5FiNSDE7gZIDa9eRXJJJ/3W0tn0x1+zfq4c+L8/bz/GYR/YfZS07W3wPaNxT6Lb4ac742GgkYYkMjVyxCMrWlG3JMLQCiNZWY4xvqGFD+6RihH7Hd196LyAq9XGw1TMyw6jaYfPR/+Z+9yb5lMVeETye/wHxvK0YScaoAW8= Content-Type: text/plain; charset="us-ascii" Content-ID: <6982FD6B53D1C3408E6DB896D0BFE060@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: d68f6b04-0c66-4109-81ce-08d6b3afd351 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Mar 2019 19:01:46.5126 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4057 Subject: Re: [dpdk-dev] [PATCH v2 3/3] net/mlx4: add secondary process support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Thu, 28 Mar 2019 19:01:48 -0000 > On Mar 26, 2019, at 12:33 PM, Shahaf Shuler <shahafs@mellanox.com> wrote: >=20 > Monday, March 25, 2019 9:18 PM, Yongseok Koh: >> To: Shahaf Shuler <shahafs@mellanox.com> >> Cc: dev@dpdk.org >> Subject: [PATCH v2 3/3] net/mlx4: add secondary process support >>=20 >> In order to support secondary process, a few features are required. >>=20 >> a) rdma-core library should allocate device resources using DPDK's memor= y >> allocator. >>=20 >> b) UAR should be remapped for secondary processes. Currently, in order n= ot >> to use different data structure for secondary processes, PMD tries to >> reserve identical virtual address space for both primary and secondary >> processes. >>=20 >> c) IPC channel is necessary, which can be easily set with rte_mp APIs. >> Through the channel, Verbs command FD is delivered to the secondary >> process and the device stop/start event is also broadcast from primary >> process. >>=20 >> Signed-off-by: Yongseok Koh <yskoh@mellanox.com> >> --- >> doc/guides/nics/features/mlx4.ini | 1 + >> doc/guides/nics/mlx4.rst | 10 + >> drivers/net/mlx4/Makefile | 6 + >> drivers/net/mlx4/meson.build | 3 + >> drivers/net/mlx4/mlx4.c | 378 >> ++++++++++++++++++++++++++++++++++++-- >> drivers/net/mlx4/mlx4.h | 60 ++++++ >> drivers/net/mlx4/mlx4_mp.c | 304 >> ++++++++++++++++++++++++++++++ >> drivers/net/mlx4/mlx4_mr.c | 32 +++- >> drivers/net/mlx4/mlx4_prm.h | 4 +- >> drivers/net/mlx4/mlx4_rxtx.c | 2 + >> drivers/net/mlx4/mlx4_rxtx.h | 1 + >> drivers/net/mlx4/mlx4_txq.c | 111 +++++++++++ >> 12 files changed, 890 insertions(+), 22 deletions(-) create mode 100644 >> drivers/net/mlx4/mlx4_mp.c >>=20 >> diff --git a/doc/guides/nics/features/mlx4.ini >> b/doc/guides/nics/features/mlx4.ini >> index a211aef332..4502aa2a87 100644 >> --- a/doc/guides/nics/features/mlx4.ini >> +++ b/doc/guides/nics/features/mlx4.ini >> @@ -29,6 +29,7 @@ Packet type parsing =3D Y >> Basic stats =3D Y >> Stats per queue =3D Y >> FW version =3D Y >> +Multiprocess aware =3D Y >> Other kdrv =3D Y >> Power8 =3D Y >> x86-32 =3D Y >> diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index >> 4ad361a2c2..cd34838f41 100644 >> --- a/doc/guides/nics/mlx4.rst >> +++ b/doc/guides/nics/mlx4.rst >> @@ -145,6 +145,16 @@ below. >> Limitations >> ----------- >>=20 >> +- For secondary process: >> + >> + - Forked secondary process not supported. >> + - All mempools must be initialized before rte_eth_dev_start(). >> + - External memory unregistered in EAL memseg list cannot be used for >> DMA >> + unless such memory has been registered by >> ``mlx4_mr_update_ext_mp()`` in >> + primary process and remapped to the same virtual address in seconda= ry >> + process. If the external memory is registered by primary process bu= t has >> + different virtual address in secondary process, unexpected error ma= y >> happen. >> + >> - CRC stripping is supported by default and always reported as "true". >> The ability to enable/disable CRC stripping requires OFED version >> 4.3-1.5.0.0 and above or rdma-core version v18 and above. >> diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile index >> b527efd625..8126b0dfc6 100644 >> --- a/drivers/net/mlx4/Makefile >> +++ b/drivers/net/mlx4/Makefile >> @@ -18,6 +18,7 @@ ifneq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y) >> SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_glue.c endif >> SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_intr.c >> +SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_mp.c >> SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_mr.c >> SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_rxq.c >> SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_rxtx.c @@ -93,6 +94,11 >> @@ mlx4_autoconf.h.new: $(RTE_SDK)/buildtools/auto-config-h.sh >> enum MLX4DV_SET_CTX_ATTR_BUF_ALLOCATORS \ >> $(AUTOCONF_OUTPUT) >> $Q sh -- '$<' '$@' \ >> + HAVE_IBV_MLX4_UAR_MMAP_OFFSET \ >> + infiniband/mlx4dv.h \ >> + enum MLX4DV_QP_MASK_UAR_MMAP_OFFSET \ >> + $(AUTOCONF_OUTPUT) >> + $Q sh -- '$<' '$@' \ >> HAVE_IBV_MLX4_WQE_LSO_SEG \ >> infiniband/mlx4dv.h \ >> type 'struct mlx4_wqe_lso_seg' \ >> diff --git a/drivers/net/mlx4/meson.build b/drivers/net/mlx4/meson.build >> index 650e2c8fbc..de020701d1 100644 >> --- a/drivers/net/mlx4/meson.build >> +++ b/drivers/net/mlx4/meson.build >> @@ -33,6 +33,7 @@ if build >> 'mlx4_ethdev.c', >> 'mlx4_flow.c', >> 'mlx4_intr.c', >> + 'mlx4_mp.c', >> 'mlx4_mr.c', >> 'mlx4_rxq.c', >> 'mlx4_rxtx.c', >> @@ -76,6 +77,8 @@ if build >> has_sym_args =3D [ >> [ 'HAVE_IBV_MLX4_BUF_ALLOCATORS', >> 'infiniband/mlx4dv.h', >> 'MLX4DV_SET_CTX_ATTR_BUF_ALLOCATORS' ], >> + [ 'HAVE_IBV_MLX4_UAR_MMAP_OFFSET', >> 'infiniband/mlx4dv.h', >> + 'MLX4DV_QP_MASK_UAR_MMAP_OFFSET' ], >> ] >> config =3D configuration_data() >> foreach arg:has_sym_args >> diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c index >> 0e0b035df0..a5cfcdbee3 100644 >> --- a/drivers/net/mlx4/mlx4.c >> +++ b/drivers/net/mlx4/mlx4.c >> @@ -17,6 +17,7 @@ >> #include <stdio.h> >> #include <stdlib.h> >> #include <string.h> >> +#include <sys/mman.h> >> #include <unistd.h> >>=20 >> /* Verbs headers do not support -pedantic. */ @@ -48,10 +49,21 @@ >> #include "mlx4_rxtx.h" >> #include "mlx4_utils.h" >>=20 >> -struct mlx4_dev_list mlx4_mem_event_cb_list =3D >> - LIST_HEAD_INITIALIZER(mlx4_mem_event_cb_list); >> +#if defined(HAVE_IBV_MLX4_UAR_MMAP_OFFSET) && \ >> + defined(HAVE_IBV_MLX4_BUF_ALLOCATORS) >> +#define HAVE_IBV_MLX4_SECONDARY_PROCESS #endif >=20 > Features should not be detected on compilation time rather by run time ba= sed on capabilities.=20 > On this case,=20 > If you are able to register the external allocator (dv call returns w/ su= ccess) and the mmap for the uar index also succeed, then you have support f= or secondary. A bit confused. Do you want to have redundant definitions in mlx5_prm.h in order to make th= e test calls? Eg., MLX4DV_SET_CTX_ATTR_BUF_ALLOCATORS and MLX4DV_QP_MASK_UAR_MMAP_OFFSET. Thanks, Yongseok From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 9F637A0679 for <public@inbox.dpdk.org>; Thu, 28 Mar 2019 20:01:50 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BF5D5493D; Thu, 28 Mar 2019 20:01:49 +0100 (CET) Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70054.outbound.protection.outlook.com [40.107.7.54]) by dpdk.org (Postfix) with ESMTP id 0F37B37A2 for <dev@dpdk.org>; Thu, 28 Mar 2019 20:01:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gknYw0YKpt3DEQOAe8slqiY6cfMaRApRIweIVLaBcGU=; b=jaqNK6OI5pinQSNzrAORqwjM27KH54GLHxg8Y8Dsof25RmDQCwweBH70J7w4MJyacCzu9lepMWFrciUUsrGQsRfnteoeCXOcvwnXQyTCEp/VOqr6dJLPL/6xo4PzPzDdchQtiM4syDPEqAUMD6npRWtJEXcPt7dl2nU+VgseogI= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4057.eurprd05.prod.outlook.com (52.134.67.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1750.17; Thu, 28 Mar 2019 19:01:46 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::6072:43be:7c2d:103a%3]) with mapi id 15.20.1750.014; Thu, 28 Mar 2019 19:01:46 +0000 From: Yongseok Koh <yskoh@mellanox.com> To: Shahaf Shuler <shahafs@mellanox.com> CC: "dev@dpdk.org" <dev@dpdk.org> Thread-Topic: [PATCH v2 3/3] net/mlx4: add secondary process support Thread-Index: AQHU4z+VDM+Spv/b9kKBel6MIVGd9qYeTykAgAMbt4A= Date: Thu, 28 Mar 2019 19:01:46 +0000 Message-ID: <41E63A87-1829-4447-AC43-09450B3AF3A7@mellanox.com> References: <20190307073905.18615-1-yskoh@mellanox.com> <20190325191801.20841-1-yskoh@mellanox.com> <20190325191801.20841-4-yskoh@mellanox.com> <AM0PR0502MB3795C2E69DB3E4A2F221461FC35F0@AM0PR0502MB3795.eurprd05.prod.outlook.com> In-Reply-To: <AM0PR0502MB3795C2E69DB3E4A2F221461FC35F0@AM0PR0502MB3795.eurprd05.prod.outlook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [69.181.245.183] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d68f6b04-0c66-4109-81ce-08d6b3afd351 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600127)(711020)(4605104)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB4057; x-ms-traffictypediagnostic: DB3PR0502MB4057: x-microsoft-antispam-prvs: <DB3PR0502MB40573C6A2E004A65143CC230C3590@DB3PR0502MB4057.eurprd05.prod.outlook.com> x-forefront-prvs: 0990C54589 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(366004)(396003)(39860400002)(376002)(136003)(189003)(199004)(14444005)(6512007)(6636002)(97736004)(186003)(53936002)(2906002)(99286004)(25786009)(478600001)(6436002)(2616005)(11346002)(6486002)(305945005)(446003)(7736002)(86362001)(6862004)(486006)(5660300002)(8936002)(106356001)(33656002)(105586002)(6246003)(36756003)(66066001)(68736007)(71200400001)(71190400001)(83716004)(53546011)(82746002)(4326008)(6506007)(26005)(476003)(102836004)(316002)(37006003)(93886005)(76176011)(14454004)(81156014)(8676002)(6116002)(3846002)(81166006)(229853002)(256004); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4057; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: xTWCGG2Jz4I/A3BJGL0pvU2jKtAiYeK0CT0lxvyItQ4Kb4+HrSdc26ANd7xjy4hCNJsT6XN+UvawpMDPtX4LTRuBmb0bv50/TlvYv69NG2TDtVCJTaHwBni9K296GzsXEY7SgqHVDsr4YFN4cJvEWQjE1Wpgo710mx0qfqD3I3LbKu2Wtha54mJnpxcIstHW6yO9CqvsgHsu21u7GH6NdQLp7jIUP2cGVjMwmtHqbAKWNGZS2gY5Z4nB+GLcexG7YAFX5FiNSDE7gZIDa9eRXJJJ/3W0tn0x1+zfq4c+L8/bz/GYR/YfZS07W3wPaNxT6Lb4ac742GgkYYkMjVyxCMrWlG3JMLQCiNZWY4xvqGFD+6RihH7Hd196LyAq9XGw1TMyw6jaYfPR/+Z+9yb5lMVeETye/wHxvK0YScaoAW8= Content-Type: text/plain; charset="UTF-8" Content-ID: <6982FD6B53D1C3408E6DB896D0BFE060@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: d68f6b04-0c66-4109-81ce-08d6b3afd351 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Mar 2019 19:01:46.5126 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4057 Subject: Re: [dpdk-dev] [PATCH v2 3/3] net/mlx4: add secondary process support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> Message-ID: <20190328190146.DoeLlN_E0eGptngtB4tRT8obIcQmzbFqrNpYIbWYYAA@z> > On Mar 26, 2019, at 12:33 PM, Shahaf Shuler <shahafs@mellanox.com> wrote: >=20 > Monday, March 25, 2019 9:18 PM, Yongseok Koh: >> To: Shahaf Shuler <shahafs@mellanox.com> >> Cc: dev@dpdk.org >> Subject: [PATCH v2 3/3] net/mlx4: add secondary process support >>=20 >> In order to support secondary process, a few features are required. >>=20 >> a) rdma-core library should allocate device resources using DPDK's memor= y >> allocator. >>=20 >> b) UAR should be remapped for secondary processes. Currently, in order n= ot >> to use different data structure for secondary processes, PMD tries to >> reserve identical virtual address space for both primary and secondary >> processes. >>=20 >> c) IPC channel is necessary, which can be easily set with rte_mp APIs. >> Through the channel, Verbs command FD is delivered to the secondary >> process and the device stop/start event is also broadcast from primary >> process. >>=20 >> Signed-off-by: Yongseok Koh <yskoh@mellanox.com> >> --- >> doc/guides/nics/features/mlx4.ini | 1 + >> doc/guides/nics/mlx4.rst | 10 + >> drivers/net/mlx4/Makefile | 6 + >> drivers/net/mlx4/meson.build | 3 + >> drivers/net/mlx4/mlx4.c | 378 >> ++++++++++++++++++++++++++++++++++++-- >> drivers/net/mlx4/mlx4.h | 60 ++++++ >> drivers/net/mlx4/mlx4_mp.c | 304 >> ++++++++++++++++++++++++++++++ >> drivers/net/mlx4/mlx4_mr.c | 32 +++- >> drivers/net/mlx4/mlx4_prm.h | 4 +- >> drivers/net/mlx4/mlx4_rxtx.c | 2 + >> drivers/net/mlx4/mlx4_rxtx.h | 1 + >> drivers/net/mlx4/mlx4_txq.c | 111 +++++++++++ >> 12 files changed, 890 insertions(+), 22 deletions(-) create mode 100644 >> drivers/net/mlx4/mlx4_mp.c >>=20 >> diff --git a/doc/guides/nics/features/mlx4.ini >> b/doc/guides/nics/features/mlx4.ini >> index a211aef332..4502aa2a87 100644 >> --- a/doc/guides/nics/features/mlx4.ini >> +++ b/doc/guides/nics/features/mlx4.ini >> @@ -29,6 +29,7 @@ Packet type parsing =3D Y >> Basic stats =3D Y >> Stats per queue =3D Y >> FW version =3D Y >> +Multiprocess aware =3D Y >> Other kdrv =3D Y >> Power8 =3D Y >> x86-32 =3D Y >> diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index >> 4ad361a2c2..cd34838f41 100644 >> --- a/doc/guides/nics/mlx4.rst >> +++ b/doc/guides/nics/mlx4.rst >> @@ -145,6 +145,16 @@ below. >> Limitations >> ----------- >>=20 >> +- For secondary process: >> + >> + - Forked secondary process not supported. >> + - All mempools must be initialized before rte_eth_dev_start(). >> + - External memory unregistered in EAL memseg list cannot be used for >> DMA >> + unless such memory has been registered by >> ``mlx4_mr_update_ext_mp()`` in >> + primary process and remapped to the same virtual address in seconda= ry >> + process. If the external memory is registered by primary process bu= t has >> + different virtual address in secondary process, unexpected error ma= y >> happen. >> + >> - CRC stripping is supported by default and always reported as "true". >> The ability to enable/disable CRC stripping requires OFED version >> 4.3-1.5.0.0 and above or rdma-core version v18 and above. >> diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile index >> b527efd625..8126b0dfc6 100644 >> --- a/drivers/net/mlx4/Makefile >> +++ b/drivers/net/mlx4/Makefile >> @@ -18,6 +18,7 @@ ifneq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y) >> SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_glue.c endif >> SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_intr.c >> +SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_mp.c >> SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_mr.c >> SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_rxq.c >> SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_rxtx.c @@ -93,6 +94,11 >> @@ mlx4_autoconf.h.new: $(RTE_SDK)/buildtools/auto-config-h.sh >> enum MLX4DV_SET_CTX_ATTR_BUF_ALLOCATORS \ >> $(AUTOCONF_OUTPUT) >> $Q sh -- '$<' '$@' \ >> + HAVE_IBV_MLX4_UAR_MMAP_OFFSET \ >> + infiniband/mlx4dv.h \ >> + enum MLX4DV_QP_MASK_UAR_MMAP_OFFSET \ >> + $(AUTOCONF_OUTPUT) >> + $Q sh -- '$<' '$@' \ >> HAVE_IBV_MLX4_WQE_LSO_SEG \ >> infiniband/mlx4dv.h \ >> type 'struct mlx4_wqe_lso_seg' \ >> diff --git a/drivers/net/mlx4/meson.build b/drivers/net/mlx4/meson.build >> index 650e2c8fbc..de020701d1 100644 >> --- a/drivers/net/mlx4/meson.build >> +++ b/drivers/net/mlx4/meson.build >> @@ -33,6 +33,7 @@ if build >> 'mlx4_ethdev.c', >> 'mlx4_flow.c', >> 'mlx4_intr.c', >> + 'mlx4_mp.c', >> 'mlx4_mr.c', >> 'mlx4_rxq.c', >> 'mlx4_rxtx.c', >> @@ -76,6 +77,8 @@ if build >> has_sym_args =3D [ >> [ 'HAVE_IBV_MLX4_BUF_ALLOCATORS', >> 'infiniband/mlx4dv.h', >> 'MLX4DV_SET_CTX_ATTR_BUF_ALLOCATORS' ], >> + [ 'HAVE_IBV_MLX4_UAR_MMAP_OFFSET', >> 'infiniband/mlx4dv.h', >> + 'MLX4DV_QP_MASK_UAR_MMAP_OFFSET' ], >> ] >> config =3D configuration_data() >> foreach arg:has_sym_args >> diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c index >> 0e0b035df0..a5cfcdbee3 100644 >> --- a/drivers/net/mlx4/mlx4.c >> +++ b/drivers/net/mlx4/mlx4.c >> @@ -17,6 +17,7 @@ >> #include <stdio.h> >> #include <stdlib.h> >> #include <string.h> >> +#include <sys/mman.h> >> #include <unistd.h> >>=20 >> /* Verbs headers do not support -pedantic. */ @@ -48,10 +49,21 @@ >> #include "mlx4_rxtx.h" >> #include "mlx4_utils.h" >>=20 >> -struct mlx4_dev_list mlx4_mem_event_cb_list =3D >> - LIST_HEAD_INITIALIZER(mlx4_mem_event_cb_list); >> +#if defined(HAVE_IBV_MLX4_UAR_MMAP_OFFSET) && \ >> + defined(HAVE_IBV_MLX4_BUF_ALLOCATORS) >> +#define HAVE_IBV_MLX4_SECONDARY_PROCESS #endif >=20 > Features should not be detected on compilation time rather by run time ba= sed on capabilities.=20 > On this case,=20 > If you are able to register the external allocator (dv call returns w/ su= ccess) and the mmap for the uar index also succeed, then you have support f= or secondary. A bit confused. Do you want to have redundant definitions in mlx5_prm.h in order to make th= e test calls? Eg., MLX4DV_SET_CTX_ATTR_BUF_ALLOCATORS and MLX4DV_QP_MASK_UAR_MMAP_OFFSET. Thanks, Yongseok