From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id DC92EA05D3 for ; Tue, 26 Mar 2019 20:33:50 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3ABE11B49A; Tue, 26 Mar 2019 20:33:50 +0100 (CET) Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70072.outbound.protection.outlook.com [40.107.7.72]) by dpdk.org (Postfix) with ESMTP id 858A61B498 for ; Tue, 26 Mar 2019 20:33:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Atxea13OiXlcu/Zi0NlZ/IvXtH4ksTkMYDY3tidgI5Q=; b=Kz+PD+SLrFfYOXt6QtCVBVYMBB9Crm4uUEPezzlWEhWmv4EtlGNz77YPpIygQEVTOGkofrWUgXBTV+JOe/O6RPt+gZRpFIqr32fZCHG/lFq9J01R+Qb9+XOOqs/gEXIRJuPS3JhWUXOj6Sw4CZ4CZasHUJJORTcyi/KWI1bjZ8Q= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by AM0PR0502MB3972.eurprd05.prod.outlook.com (52.133.36.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1750.15; Tue, 26 Mar 2019 19:33:46 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1750.014; Tue, 26 Mar 2019 19:33:46 +0000 From: Shahaf Shuler To: Yongseok Koh CC: "dev@dpdk.org" Thread-Topic: [PATCH v2 3/3] net/mlx4: add secondary process support Thread-Index: AQHU4z+MSjsDOxqcM0KS6fnD0ejQG6YeTYow Date: Tue, 26 Mar 2019 19:33:46 +0000 Message-ID: References: <20190307073905.18615-1-yskoh@mellanox.com> <20190325191801.20841-1-yskoh@mellanox.com> <20190325191801.20841-4-yskoh@mellanox.com> In-Reply-To: <20190325191801.20841-4-yskoh@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.105] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 618661dd-2dce-443a-7cb5-08d6b221f71b x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600127)(711020)(4605104)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB3972; x-ms-traffictypediagnostic: AM0PR0502MB3972: x-microsoft-antispam-prvs: x-forefront-prvs: 09888BC01D x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(376002)(136003)(396003)(346002)(366004)(189003)(199004)(26005)(6506007)(53936002)(7696005)(5660300002)(102836004)(71200400001)(55016002)(6636002)(71190400001)(9686003)(76176011)(86362001)(186003)(52536014)(229853002)(476003)(99286004)(68736007)(74316002)(316002)(486006)(11346002)(305945005)(7736002)(25786009)(14454004)(8936002)(2906002)(106356001)(8676002)(6116002)(446003)(3846002)(478600001)(256004)(33656002)(6862004)(14444005)(81156014)(6436002)(66066001)(6246003)(4326008)(97736004)(81166006)(105586002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3972; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: T7JJMN7HP4caDDe4aHGbyFHGE729GVve4SJJ5BC0olYOLesMoJS3nTKDiPrNRqUpWVNf12CfNyxGVo+cNn+Jj878R1NHR45wmFOMIbLb0IYXiy15l7T0JWfpoGNTNbmK53HBLYnSUgDwSAh9bD8rFGIucWy1qEfGAutVQrzI3R6Y4tNi6GlzJX1wmKrnGJL5Dci4PjAG3dtDeiNj4NcyQCrY6/zRRkyn74nLNTfQGbzzB+t6V8p3JPcrjN8od5bFWrCivLyMaDiqBvq5F7/VFqcXg/gS6BCvehD8F76Ro3DwdCtI+X8/B146cw3W/88005J8OHPPMDgbpuFE756jXOmuQS6WaT/7lD2MXHdvByXu81ozM2gHHgptKiOhKYWE5Qc/Xgtz9NS0H2eAKhbqiMtNk7SGjMTD67kzw2FWuJA= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 618661dd-2dce-443a-7cb5-08d6b221f71b X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Mar 2019 19:33:46.7611 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3972 Subject: Re: [dpdk-dev] [PATCH v2 3/3] net/mlx4: add secondary process support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190326193346.Zha_2Vy8rcnehkO8uMAN-ndlPVbNVNJ5J4UbehaVHJk@z> Monday, March 25, 2019 9:18 PM, Yongseok Koh: > To: Shahaf Shuler > Cc: dev@dpdk.org > Subject: [PATCH v2 3/3] net/mlx4: add secondary process support >=20 > In order to support secondary process, a few features are required. >=20 > a) rdma-core library should allocate device resources using DPDK's memory > allocator. >=20 > b) UAR should be remapped for secondary processes. Currently, in order no= t > to use different data structure for secondary processes, PMD tries to > reserve identical virtual address space for both primary and secondary > processes. >=20 > c) IPC channel is necessary, which can be easily set with rte_mp APIs. > Through the channel, Verbs command FD is delivered to the secondary > process and the device stop/start event is also broadcast from primary > process. >=20 > Signed-off-by: Yongseok Koh > --- > doc/guides/nics/features/mlx4.ini | 1 + > doc/guides/nics/mlx4.rst | 10 + > drivers/net/mlx4/Makefile | 6 + > drivers/net/mlx4/meson.build | 3 + > drivers/net/mlx4/mlx4.c | 378 > ++++++++++++++++++++++++++++++++++++-- > drivers/net/mlx4/mlx4.h | 60 ++++++ > drivers/net/mlx4/mlx4_mp.c | 304 > ++++++++++++++++++++++++++++++ > drivers/net/mlx4/mlx4_mr.c | 32 +++- > drivers/net/mlx4/mlx4_prm.h | 4 +- > drivers/net/mlx4/mlx4_rxtx.c | 2 + > drivers/net/mlx4/mlx4_rxtx.h | 1 + > drivers/net/mlx4/mlx4_txq.c | 111 +++++++++++ > 12 files changed, 890 insertions(+), 22 deletions(-) create mode 100644 > drivers/net/mlx4/mlx4_mp.c >=20 > diff --git a/doc/guides/nics/features/mlx4.ini > b/doc/guides/nics/features/mlx4.ini > index a211aef332..4502aa2a87 100644 > --- a/doc/guides/nics/features/mlx4.ini > +++ b/doc/guides/nics/features/mlx4.ini > @@ -29,6 +29,7 @@ Packet type parsing =3D Y > Basic stats =3D Y > Stats per queue =3D Y > FW version =3D Y > +Multiprocess aware =3D Y > Other kdrv =3D Y > Power8 =3D Y > x86-32 =3D Y > diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index > 4ad361a2c2..cd34838f41 100644 > --- a/doc/guides/nics/mlx4.rst > +++ b/doc/guides/nics/mlx4.rst > @@ -145,6 +145,16 @@ below. > Limitations > ----------- >=20 > +- For secondary process: > + > + - Forked secondary process not supported. > + - All mempools must be initialized before rte_eth_dev_start(). > + - External memory unregistered in EAL memseg list cannot be used for > DMA > + unless such memory has been registered by > ``mlx4_mr_update_ext_mp()`` in > + primary process and remapped to the same virtual address in secondar= y > + process. If the external memory is registered by primary process but= has > + different virtual address in secondary process, unexpected error may > happen. > + > - CRC stripping is supported by default and always reported as "true". > The ability to enable/disable CRC stripping requires OFED version > 4.3-1.5.0.0 and above or rdma-core version v18 and above. > diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile index > b527efd625..8126b0dfc6 100644 > --- a/drivers/net/mlx4/Makefile > +++ b/drivers/net/mlx4/Makefile > @@ -18,6 +18,7 @@ ifneq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y) > SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_glue.c endif > SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_intr.c > +SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_mp.c > SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_mr.c > SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_rxq.c > SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) +=3D mlx4_rxtx.c @@ -93,6 +94,11 > @@ mlx4_autoconf.h.new: $(RTE_SDK)/buildtools/auto-config-h.sh > enum MLX4DV_SET_CTX_ATTR_BUF_ALLOCATORS \ > $(AUTOCONF_OUTPUT) > $Q sh -- '$<' '$@' \ > + HAVE_IBV_MLX4_UAR_MMAP_OFFSET \ > + infiniband/mlx4dv.h \ > + enum MLX4DV_QP_MASK_UAR_MMAP_OFFSET \ > + $(AUTOCONF_OUTPUT) > + $Q sh -- '$<' '$@' \ > HAVE_IBV_MLX4_WQE_LSO_SEG \ > infiniband/mlx4dv.h \ > type 'struct mlx4_wqe_lso_seg' \ > diff --git a/drivers/net/mlx4/meson.build b/drivers/net/mlx4/meson.build > index 650e2c8fbc..de020701d1 100644 > --- a/drivers/net/mlx4/meson.build > +++ b/drivers/net/mlx4/meson.build > @@ -33,6 +33,7 @@ if build > 'mlx4_ethdev.c', > 'mlx4_flow.c', > 'mlx4_intr.c', > + 'mlx4_mp.c', > 'mlx4_mr.c', > 'mlx4_rxq.c', > 'mlx4_rxtx.c', > @@ -76,6 +77,8 @@ if build > has_sym_args =3D [ > [ 'HAVE_IBV_MLX4_BUF_ALLOCATORS', > 'infiniband/mlx4dv.h', > 'MLX4DV_SET_CTX_ATTR_BUF_ALLOCATORS' ], > + [ 'HAVE_IBV_MLX4_UAR_MMAP_OFFSET', > 'infiniband/mlx4dv.h', > + 'MLX4DV_QP_MASK_UAR_MMAP_OFFSET' ], > ] > config =3D configuration_data() > foreach arg:has_sym_args > diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c index > 0e0b035df0..a5cfcdbee3 100644 > --- a/drivers/net/mlx4/mlx4.c > +++ b/drivers/net/mlx4/mlx4.c > @@ -17,6 +17,7 @@ > #include > #include > #include > +#include > #include >=20 > /* Verbs headers do not support -pedantic. */ @@ -48,10 +49,21 @@ > #include "mlx4_rxtx.h" > #include "mlx4_utils.h" >=20 > -struct mlx4_dev_list mlx4_mem_event_cb_list =3D > - LIST_HEAD_INITIALIZER(mlx4_mem_event_cb_list); > +#if defined(HAVE_IBV_MLX4_UAR_MMAP_OFFSET) && \ > + defined(HAVE_IBV_MLX4_BUF_ALLOCATORS) > +#define HAVE_IBV_MLX4_SECONDARY_PROCESS #endif Features should not be detected on compilation time rather by run time base= d on capabilities.=20 On this case,=20 If you are able to register the external allocator (dv call returns w/ succ= ess) and the mmap for the uar index also succeed, then you have support for= secondary.