From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id BE161A04B6; Sun, 11 Oct 2020 16:03:24 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 37CF31D571; Sun, 11 Oct 2020 16:03:23 +0200 (CEST) Received: from hqnvemgate24.nvidia.com (hqnvemgate24.nvidia.com [216.228.121.143]) by dpdk.org (Postfix) with ESMTP id 9E05B1D447 for ; Sun, 11 Oct 2020 16:03:21 +0200 (CEST) Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Sun, 11 Oct 2020 07:01:26 -0700 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Sun, 11 Oct 2020 14:03:19 +0000 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (104.47.74.47) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Sun, 11 Oct 2020 14:03:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ELOmmJKWuW/ShT7Ah/Zkpev3K7A13mHJVByHFxdcK5YB9yuaWTw08Vfl56GxHWVURpGAn/A/cGMgv9r7XNSxk8rVjOnda7PrkP4KN2ZAWMcaD9cwGvGgvlA0Q5lpzH1nA0YoIwlKre6eDbNjGgJSA9gpsAj3FlIZ9hWJlRHOKcGD1c8FUniNJU8sVBXQaAt0X69bhEIp9UfGmOKyusKCPKkgdn3AgEQOB4Gm3/gsb33wZ5pKuQYI0eNbLCkN6r/eFt98GB4aWCqGSE/9v7ToSvbzLGnevho8GPT4Psw3q6tVae/zYqOKy6RMP8sGeu6j7Qq2QwGWc4o5ZD1Z6p1uDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rhbg2RdWpqRNrfOPvDNeZqNqaTWgMdckWJVoSLKZZjU=; b=meGBWxVQ4XknxIdAtXgTqMOIaX88LMTlLZkC8SiUHSZePKNrW+97guKXXAApBn8gr17KzMClM6iEVVmQ2J4mzn5GoTXDgrxDn8cBkgElnmf8wCvk9I8w2NvUh69vzj7GN6KvAxhc4+Yo7ln8bPLUzbYhAHy0th9HwYgA9XOpuN4Dom3I7GBQaq0EnLzqo2CE02ZstchqP94/XTYXOHab3Ty+deTyF/JyHp4IQLzzkr1xsVB5uqPgkpnKpbI+5a7M0/q/bliAOrw4pMo8+tGxwWxyxLCcfjrnvDoE/lgyAOwyBR8d0E2/eqx2c1XdFYFlGvUyNZT92M/9FsYwe2bi6w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Received: from MN2PR12MB4286.namprd12.prod.outlook.com (2603:10b6:208:199::22) by BL0PR12MB4692.namprd12.prod.outlook.com (2603:10b6:208:82::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3455.23; Sun, 11 Oct 2020 14:03:17 +0000 Received: from MN2PR12MB4286.namprd12.prod.outlook.com ([fe80::61fd:a36e:cf4f:2d3f]) by MN2PR12MB4286.namprd12.prod.outlook.com ([fe80::61fd:a36e:cf4f:2d3f%9]) with mapi id 15.20.3455.029; Sun, 11 Oct 2020 14:03:17 +0000 From: Ori Kam To: Bing Zhao , "viacheslavo@mellanox.com" , "matan@mellanox.com" CC: "dev@dpdk.org" , Raslan Darawsheh Thread-Topic: [dpdk-dev] [PATCH] net/mlx5: add flow sync API Thread-Index: AQHWnk0kCZxIrSVCjkG62qTNvW8t+qmScBdQ Date: Sun, 11 Oct 2020 14:03:17 +0000 Message-ID: References: <1602255678-108560-1-git-send-email-bingz@nvidia.com> In-Reply-To: <1602255678-108560-1-git-send-email-bingz@nvidia.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: nvidia.com; dkim=none (message not signed) header.d=none;nvidia.com; dmarc=none action=none header.from=nvidia.com; x-originating-ip: [147.236.152.129] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: c614e20a-dc82-4a88-5984-08d86dee671d x-ms-traffictypediagnostic: BL0PR12MB4692: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6430; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: cd2TwAkeTwOncxUPfkf6WvolW9ywvG/jQLXrgUCbQO1L9TmnQ9QwqqYpWEDNfW0lMsnEohTF95boaT7Z8/xWeNCt40KHJ5HJkOW7kk0KTRF6Mt7P4Z+ACiYMoOG9RGsFL++CdMQsZRLNW9RQMObAFoHRWhbO07CccM1Ckx+d5EBDnPpC4UTPUZc+xQDXS9IbbPzk62e6TTgM0XHOZiPn9uAulIYb8f6YHhOWatN3LRCEZ9tukUSdzP/EqtxRpjUfK53LrQ3jwVeVAmgYNW7Wpt7WvLRLMYMGA2MlB4X8cgZI4+Vv1OhKsNHHiYPJui0SV8ELwY0lEemmLd7sZLZXdA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR12MB4286.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(376002)(39860400002)(346002)(396003)(136003)(110136005)(54906003)(316002)(52536014)(478600001)(4326008)(8676002)(6506007)(8936002)(9686003)(107886003)(71200400001)(2906002)(66946007)(66446008)(86362001)(83380400001)(5660300002)(26005)(186003)(53546011)(33656002)(76116006)(66556008)(7696005)(55016002)(66476007)(64756008); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: hVuY45gTNIsPYfA6O8Dm6mUwzj794cTlX+4gLsnBqRJsedL/CXhObDK8e8DHvKL/G8F6x/gPnXzeuXwEl/jjLRvxKQo94skYD5Tjq973VSCBPcH/P8RGOgPu/l4In7NefuffwU6lhD5Te5BOYeyWIiruDDM8Cpa08j3KpAWmdkVnc51SmR5xTwKWoQf2L7lKA8gpbq2+AF/BH8HX7ITWq/zGwn3WHpve5IhbrAeaQlN2qUBt4DhhiiILm8MsXSWHwoPXIMj++A0LQ9Vwcdm/poPvBrFUkpfMCHf6mMz7Ub0PseBTDUdKZYBF02VqA9mpDCenv/mtcJbyhc14AqR7rP7Tw76XTcOHjg6C49g8zFv3YIgoPAiSaS0RTR2eR29wdKO6vydTuPzg9W5ViaJ4cLF2wvLc7IT9IeQGFikCDSAbHHU3+Q5w3iIguCFSME1UD7dQq6eIATUk5BiR++9kJa/yAgY32Y+GNwYHk2bvhfFgW9b9dCP/8uSwvG/0w8ZcbSpLbFtUCwL5zIpJx1dDOZyloqwKPT7iSg3R0JjgKxf5f/1h8butTOB0nTntFrahjL+961aUe0SyB17kpiG41v9u0YZ3J0VfIRdt4SXBjkV+H0TDRip9pTmFuFVqAKqq2jrJZ8XR2wDWfRU0eTWp7w== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4286.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: c614e20a-dc82-4a88-5984-08d86dee671d X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Oct 2020 14:03:17.0534 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: wP7X/xkEPPNuwgf71jVW2EFh3pUpKD3rTn2vrV2nGzp6XtQaGxZ2VN1LadDpoMnmYtQSYe4T+WiZo0nD9YAJUQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4692 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1602424886; bh=rhbg2RdWpqRNrfOPvDNeZqNqaTWgMdckWJVoSLKZZjU=; h=ARC-Seal:ARC-Message-Signature:ARC-Authentication-Results:From:To: CC:Subject:Thread-Topic:Thread-Index:Date:Message-ID:References: In-Reply-To:Accept-Language:Content-Language:X-MS-Has-Attach: X-MS-TNEF-Correlator:authentication-results:x-originating-ip: x-ms-publictraffictype:x-ms-office365-filtering-correlation-id: x-ms-traffictypediagnostic:x-ms-exchange-transport-forked: x-microsoft-antispam-prvs:x-ms-oob-tlc-oobclassifiers: x-ms-exchange-senderadcheck:x-microsoft-antispam: x-microsoft-antispam-message-info:x-forefront-antispam-report: x-ms-exchange-antispam-messagedata:Content-Type: Content-Transfer-Encoding:MIME-Version: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-originalarrivaltime: X-MS-Exchange-CrossTenant-fromentityheader: X-MS-Exchange-CrossTenant-id:X-MS-Exchange-CrossTenant-mailboxtype: X-MS-Exchange-CrossTenant-userprincipalname: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=A/TdIivx4yQ2nrTgBJ7L5wWubTkby3/TcLAEpy3zGtR1Z13LX2taISbqjjCL5d4oC Aa1kKd+6gyswhRJvOMRikkt42xJEjuo4IbPl/kh9kC0tBjcFuGB11+DaktClyFY/Sp 2vggKBp61MQ+5IDuPTv7f1g2+SVrs84wqOzoeonSrHluUsP5Dap17eNeia3hvBgmpV 5QH3hpEuvaxQ0DY1sskyXxw/t+WRfYpEIm8zgJMerLtc9xGpkgz6dWsk6A4jarnM3+ lrW1EjU87vyzZLFfT3tSpcC/x4cOlwLU7Lw08mdrSRMrHu+ZT//r/dLkFz0PPoQ2yu 8432shB463zhQ== Subject: Re: [dpdk-dev] [PATCH] net/mlx5: add flow sync API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Bing, > -----Original Message----- > From: dev On Behalf Of Bing Zhao > Sent: Friday, October 9, 2020 6:01 PM > Subject: [dpdk-dev] [PATCH] net/mlx5: add flow sync API >=20 > When creating a flow, the rule itself might not take effort > immediately once the function call returns with success. It would > take some time to let the steering synchronize with the hardware. >=20 > If the application wants the packet to be sent to hit the flow after > it it created, this flow sync API can be used to clear the steering > HW cache to enforce next packet hits the latest rules. >=20 > For TX, usually the NIC TX domain and/or the FDB domain should be > synchronized depends in which domain the flow is created. >=20 > The application could also try to synchronize the NIC RX and/or the > FDB domain for the ingress packets. But in the real life, it is hard > to determine when a packet will come into the NIC. >=20 > Signed-off-by: Bing Zhao > --- > drivers/common/mlx5/linux/mlx5_glue.c | 14 ++++++++++++++ > drivers/common/mlx5/linux/mlx5_glue.h | 1 + > drivers/net/mlx5/mlx5_flow.c | 22 ++++++++++++++++++++++ > drivers/net/mlx5/mlx5_flow.h | 5 +++++ > drivers/net/mlx5/mlx5_flow_dv.c | 25 +++++++++++++++++++++++++ > drivers/net/mlx5/rte_pmd_mlx5.h | 19 +++++++++++++++++++ > drivers/net/mlx5/rte_pmd_mlx5_version.map | 2 ++ > 7 files changed, 88 insertions(+) >=20 Missing release notes and mlx5.rst. > diff --git a/drivers/common/mlx5/linux/mlx5_glue.c > b/drivers/common/mlx5/linux/mlx5_glue.c > index fcf03e8..86047b1 100644 > --- a/drivers/common/mlx5/linux/mlx5_glue.c > +++ b/drivers/common/mlx5/linux/mlx5_glue.c > @@ -494,6 +494,19 @@ > #endif > } >=20 > +static int > +mlx5_glue_dr_sync_domain(void *domain, uint32_t flags) > +{ > +#ifdef HAVE_MLX5DV_DR > + return mlx5dv_dr_domain_sync(domain, flags); > +#else > + (void)domain; > + (void)flags; > + errno =3D ENOTSUP; > + return errno; > +#endif > +} > + > static struct ibv_cq_ex * > mlx5_glue_dv_create_cq(struct ibv_context *context, > struct ibv_cq_init_attr_ex *cq_attr, > @@ -1298,6 +1311,7 @@ > .dr_destroy_flow_tbl =3D mlx5_glue_dr_destroy_flow_tbl, > .dr_create_domain =3D mlx5_glue_dr_create_domain, > .dr_destroy_domain =3D mlx5_glue_dr_destroy_domain, > + .dr_sync_domain =3D mlx5_glue_dr_sync_domain, > .dv_create_cq =3D mlx5_glue_dv_create_cq, > .dv_create_wq =3D mlx5_glue_dv_create_wq, > .dv_query_device =3D mlx5_glue_dv_query_device, > diff --git a/drivers/common/mlx5/linux/mlx5_glue.h > b/drivers/common/mlx5/linux/mlx5_glue.h > index 734ace2..d24a16e 100644 > --- a/drivers/common/mlx5/linux/mlx5_glue.h > +++ b/drivers/common/mlx5/linux/mlx5_glue.h > @@ -195,6 +195,7 @@ struct mlx5_glue { > void *(*dr_create_domain)(struct ibv_context *ctx, > enum mlx5dv_dr_domain_type domain); > int (*dr_destroy_domain)(void *domain); > + int (*dr_sync_domain)(void *domain, uint32_t flags); > struct ibv_cq_ex *(*dv_create_cq) > (struct ibv_context *context, > struct ibv_cq_init_attr_ex *cq_attr, > diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c > index a94f630..e25ec0c 100644 > --- a/drivers/net/mlx5/mlx5_flow.c > +++ b/drivers/net/mlx5/mlx5_flow.c > @@ -29,6 +29,7 @@ > #include "mlx5_flow.h" > #include "mlx5_flow_os.h" > #include "mlx5_rxtx.h" > +#include "rte_pmd_mlx5.h" >=20 > /** Device flow drivers. */ > extern const struct mlx5_flow_driver_ops mlx5_flow_verbs_drv_ops; > @@ -6310,3 +6311,24 @@ struct mlx5_meter_domains_infos * > dev->data->port_id); > return -ENOTSUP; > } > + > +static int > +mlx5_flow_sync_memory(struct rte_eth_dev *dev, uint32_t domains, uint32_= t > flags) > +{ > + const struct mlx5_flow_driver_ops *fops; > + struct rte_flow_attr attr =3D { .transfer =3D 0 }; > + > + if (flow_get_drv_type(dev, &attr) =3D=3D MLX5_FLOW_TYPE_DV) { > + fops =3D flow_get_drv_ops(MLX5_FLOW_TYPE_DV); > + return fops->sync_memory(dev, domains, flags); > + } > + return -ENOTSUP; > +} > + > +int rte_pmd_mlx5_sync_flow(uint16_t port_id, uint32_t domains) > +{ > + struct rte_eth_dev *dev =3D &rte_eth_devices[port_id]; > + > + return mlx5_flow_sync_memory(dev, domains, > + MLX5DV_DR_DOMAIN_SYNC_FLAGS_HW); > +} > diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h > index 279daf2..ae0a508 100644 > --- a/drivers/net/mlx5/mlx5_flow.h > +++ b/drivers/net/mlx5/mlx5_flow.h > @@ -910,6 +910,10 @@ typedef int (*mlx5_flow_get_aged_flows_t) > void **context, > uint32_t nb_contexts, > struct rte_flow_error *error); > +typedef int (*mlx5_flow_sync_memory_t) > + (struct rte_eth_dev *dev, > + uint32_t domains, > + uint32_t flags); > struct mlx5_flow_driver_ops { > mlx5_flow_validate_t validate; > mlx5_flow_prepare_t prepare; > @@ -926,6 +930,7 @@ struct mlx5_flow_driver_ops { > mlx5_flow_counter_free_t counter_free; > mlx5_flow_counter_query_t counter_query; > mlx5_flow_get_aged_flows_t get_aged_flows; > + mlx5_flow_sync_memory_t sync_memory; > }; >=20 > /* mlx5_flow.c */ > diff --git a/drivers/net/mlx5/mlx5_flow_dv.c > b/drivers/net/mlx5/mlx5_flow_dv.c > index 79fdf34..b78ffc5 100644 > --- a/drivers/net/mlx5/mlx5_flow_dv.c > +++ b/drivers/net/mlx5/mlx5_flow_dv.c > @@ -10150,6 +10150,30 @@ struct field_modify_info modify_tcp[] =3D { > flow_dv_shared_unlock(dev); > } >=20 > +static int > +flow_dv_sync_domain(struct rte_eth_dev *dev, uint32_t domains, uint32_t > flags) > +{ > + struct mlx5_priv *priv =3D dev->data->dev_private; > + int ret =3D 0; > + > + if (domains & (1 << MLX5DV_FLOW_TABLE_TYPE_NIC_RX)) { Shouldn't this value be MLX5DV_DR_DOMAIN_TYPE_NIC_RX? > + ret =3D mlx5_glue->dr_sync_domain(priv->sh->rx_domain, flags); > + if (ret) > + return ret; > + } > + if (domains & (1 << MLX5DV_FLOW_TABLE_TYPE_NIC_TX)) { > + ret =3D mlx5_glue->dr_sync_domain(priv->sh->tx_domain, flags); > + if (ret) > + return ret; > + } > + if (domains & (1 << MLX5DV_FLOW_TABLE_TYPE_FDB)) { > + ret =3D mlx5_glue->dr_sync_domain(priv->sh->fdb_domain, > flags); > + if (ret) > + return ret; > + } > + return 0; > +} > + > const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops =3D { > .validate =3D flow_dv_validate, > .prepare =3D flow_dv_prepare, > @@ -10166,6 +10190,7 @@ struct field_modify_info modify_tcp[] =3D { > .counter_free =3D flow_dv_counter_free, > .counter_query =3D flow_dv_counter_query, > .get_aged_flows =3D flow_get_aged_flows, > + .sync_memory =3D flow_dv_sync_domain, > }; >=20 > #endif /* HAVE_IBV_FLOW_DV_SUPPORT */ > diff --git a/drivers/net/mlx5/rte_pmd_mlx5.h > b/drivers/net/mlx5/rte_pmd_mlx5.h > index 8c69228..636dd07 100644 > --- a/drivers/net/mlx5/rte_pmd_mlx5.h > +++ b/drivers/net/mlx5/rte_pmd_mlx5.h > @@ -32,4 +32,23 @@ > __rte_experimental > int rte_pmd_mlx5_get_dyn_flag_names(char *names[], unsigned int n); >=20 > +/** > + * Synchronize the flows to make them take effort on hardware. > + * > + * @param[in] port_id > + * The port identifier of the Ethernet device.. > + * @param[in] domains > + * Bitmask of domains in which synchronization will be done. > + * Refer to "/usr/include/infiniband/mlx5dv.h" > + * The index of bit that set represents the corresponding domain ID. > + * I think it will be good to state the enum name, Just to make sure I understand the domain value for FDB should be: (1 << MLX5DV_DR_DOMAIN_TYPE_FDB), am I correct? > + * @return > + * - (0) if successful. > + * - (-EINVAL) if bad parameter. > + * - (-ENOTSUP) if hardware doesn't support. > + * - Other errors > + */ > +__rte_experimental > +int rte_pmd_mlx5_sync_flow(uint16_t port_id, uint32_t domains); > + > #endif > diff --git a/drivers/net/mlx5/rte_pmd_mlx5_version.map > b/drivers/net/mlx5/rte_pmd_mlx5_version.map > index bc1d3d0..82a32b5 100644 > --- a/drivers/net/mlx5/rte_pmd_mlx5_version.map > +++ b/drivers/net/mlx5/rte_pmd_mlx5_version.map > @@ -7,4 +7,6 @@ EXPERIMENTAL { >=20 > # added in 20.02 > rte_pmd_mlx5_get_dyn_flag_names; > + # added in 20.11 > + rte_pmd_mlx5_sync_flow; > }; > -- > 1.8.3.1