From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20063.outbound.protection.outlook.com [40.107.2.63]) by dpdk.org (Postfix) with ESMTP id 3BFF61B173 for ; Fri, 28 Sep 2018 01:40:13 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OGVRqWWLbOOTKckxusBDxhl4JzghRf46zT8DIf6e/4c=; b=pyLCaBwsVheoTJwbmU88/YgJ3ZdYH/8QhRvlRbDO+BmExfAh1KooOfW1i7nlu46JE5mJ7u/I8DTiAqnBjFRB5nFCDQ3HWlUZQshE6tfims/bEL0ccwa2K1p5StaIk9ubLBnaN3+dlPU6NqQBGmUFlQETTQQCtgz2uRu30rUYkyM= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB3979.eurprd05.prod.outlook.com (52.134.67.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1164.25; Thu, 27 Sep 2018 23:40:10 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::1cb0:661b:ecab:6045]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::1cb0:661b:ecab:6045%2]) with mapi id 15.20.1164.024; Thu, 27 Sep 2018 23:40:10 +0000 From: Yongseok Koh To: Slava Ovsiienko CC: dev , Shahaf Shuler Thread-Topic: [dpdk-dev] [RFC] mlx5: flow counters support on the linux-rdma v19 base Thread-Index: AQHUOIqYUzqYQ/0JxEm+GOHQEEC3Y6UFBcWA Date: Thu, 27 Sep 2018 23:40:10 +0000 Message-ID: References: <1534772079-6514-1-git-send-email-viacheslavo@mellanox.com> In-Reply-To: <1534772079-6514-1-git-send-email-viacheslavo@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [69.181.245.183] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB3PR0502MB3979; 6:wNCOIwIIzE4evKhAoI89LmTKVgxLEeVxCLq/9cOL+KbVzZrcDp57w2tKclOOPOeIGLGz5M9toLPFBM3wNYHKeAHfYgGl9E8codQ3sjJkmzYRJ22POKvCufEAgGA38HoFq174AyKkeImwJ3fsMIL5a984KiYYvhu18tpva5f5DIb2IIQf3qpVQkCap9RoYToNDlt2wfsxYxPcJ3RnWtMV0AxV2LO/J00X9ujhNNJVar7CN6lo5iIBJjetnjvzEMn47PTfJLkw9jHHoBy2ZrOMSKMvZtiD6eKmrxHo3QKubnSs+DwyidK+AO5mVypftJ2y2oLg31BODC+jvDC9uzficwkuMXJuIaaoYKO/MHXYTTo6/XILr38XEspPp8uvh913m1sft06B36GT2n/5SXj/bE6FhFxKY+ibn2yit3/vreMWnAbajYq1ZOnZUiZ2+WIyvTaoOqUTQgqrnWGY4gMBkg==; 5:Jj6L4Fon6tzTeA6DCtUZcDixVK9nATO011jVoVaw7CDoZ7SEgAqxGEYibNaKgDFARlYshlNjInG7WnHHwQoZxbIFNVnvuBvgpMYiHOA9Lr2AJCj8AZ/gx2qMdOBJ4ynzEfyS6lvGuvHq9QhnZAa6ilr0HNl7badMSe8BpP52lIM=; 7:vKDW/P9HgBH+5FgbAg5H78L02rTf3TkzKCuHCQX5ZGsr1aeMlw3J4W4nvC0EhOs04sET4m3ORu2Z5TMXQo8R4V4WLZA0fz4yy0dAvBrZcVutilo13Tqi+1//s0uqn/UP6DBooRrnT6f1QD/daYOMXD6fnvIZ48tZBW4gu+ajtzb1DyzmXFmPJ+RFvzp2aQ2oCZ2VVBsjy1QkdWQXim2OD7adVWKCYcQ0VTNj2Z2nwz3lqpdvp18u0+NWDLHg7Vym x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 585760ee-0370-46fd-2c12-08d624d29073 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB3979; x-ms-traffictypediagnostic: DB3PR0502MB3979: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3231355)(944501410)(52105095)(3002001)(6055026)(149066)(150057)(6041310)(20161123562045)(20161123558120)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(201708071742011)(7699051); SRVR:DB3PR0502MB3979; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB3979; x-forefront-prvs: 0808323E97 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(136003)(396003)(366004)(376002)(39860400002)(189003)(199004)(6486002)(5250100002)(6436002)(2906002)(66066001)(229853002)(2900100001)(25786009)(6512007)(6636002)(36756003)(6862004)(478600001)(107886003)(6246003)(53936002)(4326008)(14454004)(5660300001)(3846002)(6116002)(102836004)(34290500001)(83716004)(71200400001)(71190400001)(256004)(26005)(82746002)(305945005)(81166006)(68736007)(81156014)(446003)(486006)(11346002)(476003)(105586002)(14444005)(8936002)(53546011)(6506007)(5024004)(99286004)(86362001)(37006003)(106356001)(2616005)(7736002)(76176011)(97736004)(54906003)(316002)(33656002)(309714004); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB3979; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: zRSeCq8g+LN9MD72TOsSDlEGf20O1nNwOWi3OWNlW+R7jYkDNZWtzGBIIQ6n4bH1C4Y1oUC0dYPC23n1ldPnKRh1gYlerAEWmCQk9vpSyiXN8QMJPn8AP4KbiCK0Xvm3kfryaNvQXxjMF5HzilTrzRgQoqISmzEDC7wMogewC9tJOrJKjXNktzx1ITA01otNDSW10HK2SezNeaC9dewFuYenmYgjld+ik9ZGp40AJdKBVKrQR4KRk3b0buZHyxfL/dXvs5PIl8g1a9aN15e7JHsNiH2lMTbYbT3cU29tG7uxcqdHUJQjlg84I+Hm118/MF7rouaHLQQYHL4sO0OhKIZWVTMRoaU6hOyrreUIbzY= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: <7E797F0F076E344EBCC043066C41F156@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 585760ee-0370-46fd-2c12-08d624d29073 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Sep 2018 23:40:10.4077 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB3979 Subject: Re: [dpdk-dev] [RFC] mlx5: flow counters support on the linux-rdma v19 base X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Sep 2018 23:40:13 -0000 > On Aug 20, 2018, at 6:34 AM, viacheslavo wrote= : >=20 > Mellanox mlx5 PMD supports Flow Counters via Verbs library. > The current implementation is based on the Mellanox proprietary > Verbs library included in MLNX OFED packages. The Flow Counter > support is recently added into linux-rdma release (v19), > so the mlx5 PMD update is needed to provide Counter feature > on the base of linux-rdma. >=20 > mlx5 PMD can be compiled with MLNX OFED or linux-rdma v19 > and provide flow counters for both. >=20 > Signed-off-by: Viacheslav Ovsiienko > --- > drivers/net/mlx5/Makefile | 10 +++++++ > drivers/net/mlx5/mlx5.c | 6 ++++ > drivers/net/mlx5/mlx5_flow.c | 67 ++++++++++++++++++++++++++++++++++++---= ----- > drivers/net/mlx5/mlx5_glue.c | 41 +++++++++++++++++++++++++++ > drivers/net/mlx5/mlx5_glue.h | 16 +++++++++++ > 5 files changed, 128 insertions(+), 12 deletions(-) >=20 > diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile > index 2e70dec..ca4c143 100644 > --- a/drivers/net/mlx5/Makefile > +++ b/drivers/net/mlx5/Makefile > @@ -155,6 +155,16 @@ mlx5_autoconf.h.new: $(RTE_SDK)/buildtools/auto-conf= ig-h.sh > type 'struct ibv_counter_set_init_attr' \ > $(AUTOCONF_OUTPUT) > $Q sh -- '$<' '$@' \ > + HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT \ > + infiniband/verbs.h \ > + type 'struct ibv_counters_init_attr' \ > + $(AUTOCONF_OUTPUT) > + $Q sh -- '$<' '$@' \ > + HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 \ > + infiniband/verbs.h \ > + type 'struct ibv_counters_init_attr' \ > + $(AUTOCONF_OUTPUT) > + $Q sh -- '$<' '$@' \ I don't see any difference between the two - HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT and HAVE_IBV_DEVICE_COUNTERS_SET_SUPPO= RT_V45?? Thanks Yongseok > HAVE_RDMA_NL_NLDEV \ > rdma/rdma_netlink.h \ > enum RDMA_NL_NLDEV \ > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c > index ec63bc6..ad289f0 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -730,8 +730,10 @@ > unsigned int mprq_min_stride_num_n =3D 0; > unsigned int mprq_max_stride_num_n =3D 0; > #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT > +#ifndef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > struct ibv_counter_set_description cs_desc =3D { .counter_type =3D 0 }; > #endif > +#endif > struct ether_addr mac; > char name[RTE_ETH_NAME_MAX_LEN]; > int own_domain_id =3D 0; > @@ -1001,11 +1003,15 @@ > DRV_LOG(DEBUG, "checksum offloading is %ssupported", > (config.hw_csum ? "" : "not ")); > #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT > +#ifndef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > config.flow_counter_en =3D !!attr.max_counter_sets; > mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); > DRV_LOG(DEBUG, "counter type =3D %d, num of cs =3D %ld, attributes =3D %= d", > cs_desc.counter_type, cs_desc.num_of_cs, > cs_desc.attributes); > +#else > + config.flow_counter_en =3D 1; > +#endif > #endif > config.ind_table_max_size =3D > attr.rss_caps.max_rwq_indirection_table_size; > diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c > index ca4625b..4d762dd 100644 > --- a/drivers/net/mlx5/mlx5_flow.c > +++ b/drivers/net/mlx5/mlx5_flow.c > @@ -290,7 +290,11 @@ struct mlx5_flow_counter { > uint32_t shared:1; /**< Share counter ID with other flow rules. */ > uint32_t ref_cnt:31; /**< Reference counter. */ > uint32_t id; /**< Counter ID. */ > - struct ibv_counter_set *cs; /**< Holds the counters for the rule. */ > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > +struct ibv_counters *cs; /**< Holds the counters for the rule. */ > +#else > +struct ibv_counter_set *cs; /**< Holds the counters for the rule. */ > +#endif > uint64_t hits; /**< Number of packets matched by the rule. */ > uint64_t bytes; /**< Number of bytes matched by the rule. */ > }; > @@ -519,27 +523,32 @@ struct mlx5_flow_tunnel_info { > static struct mlx5_flow_counter * > mlx5_flow_counter_new(struct rte_eth_dev *dev, uint32_t shared, uint32_t = id) > { > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT > struct priv *priv =3D dev->data->dev_private; > struct mlx5_flow_counter *cnt; >=20 > - LIST_FOREACH(cnt, &priv->flow_counters, next) { > - if (!cnt->shared || cnt->shared !=3D shared) > - continue; > - if (cnt->id !=3D id) > - continue; > - cnt->ref_cnt++; > - return cnt; > + if (shared) { > + LIST_FOREACH(cnt, &priv->flow_counters, next) > + if (cnt->shared && cnt->id =3D=3D id) { > + cnt->ref_cnt++; > + return cnt; > + } > } > -#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT >=20 > struct mlx5_flow_counter tmpl =3D { > .shared =3D shared, > .id =3D id, > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + .cs =3D mlx5_glue->create_counter_set > + (priv->ctx, > + &(struct ibv_counters_init_attr){0}), > +#else > .cs =3D mlx5_glue->create_counter_set > (priv->ctx, > &(struct ibv_counter_set_init_attr){ > .counter_set_id =3D id, > }), > +#endif > .hits =3D 0, > .bytes =3D 0, > }; > @@ -548,17 +557,40 @@ struct mlx5_flow_tunnel_info { > rte_errno =3D errno; > return NULL; > } > + > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + struct ibv_counter_attach_attr attach_attr =3D {0}; > + int ret; > + > + attach_attr.counter_desc =3D IBV_COUNTER_PACKETS; > + attach_attr.index =3D 0; > + ret =3D ibv_attach_counters_point_flow(tmpl.cs, &attach_attr, NULL); > + if (!ret) { > + attach_attr.counter_desc =3D IBV_COUNTER_BYTES; > + attach_attr.index =3D 1; > + ret =3D ibv_attach_counters_point_flow(tmpl.cs, > + &attach_attr, > + NULL); > + } > + if (ret) { > + claim_zero(mlx5_glue->destroy_counter_set(tmpl.cs)); > + rte_errno =3D ret; > + return NULL; > + } > +#endif > cnt =3D rte_calloc(__func__, 1, sizeof(*cnt), 0); > if (!cnt) { > + claim_zero(mlx5_glue->destroy_counter_set(tmpl.cs)); > rte_errno =3D ENOMEM; > return NULL; > } > *cnt =3D tmpl; > LIST_INSERT_HEAD(&priv->flow_counters, cnt, next); > return cnt; > -#endif > +#else > rte_errno =3D ENOTSUP; > return NULL; > +#endif > } >=20 > /** > @@ -2307,7 +2339,7 @@ struct mlx5_flow_tunnel_info { > flow->counter =3D mlx5_flow_counter_new(dev, count->shared, > count->id); > if (!flow->counter) > - return rte_flow_error_set(error, ENOTSUP, > + return rte_flow_error_set(error, rte_errno, > RTE_FLOW_ERROR_TYPE_ACTION, > action, > "cannot get counter" > @@ -2320,7 +2352,11 @@ struct mlx5_flow_tunnel_info { > "flow counters are not supported."); > flow->modifier |=3D MLX5_FLOW_MOD_COUNT; > #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + counter.counters =3D flow->counter->cs; > +#else > counter.counter_set_handle =3D flow->counter->cs->handle; > +#endif > if (size <=3D flow_size) > mlx5_flow_spec_verbs_add(flow, &counter, size); > return size; > @@ -3373,6 +3409,13 @@ struct rte_flow * > if (flow->modifier & MLX5_FLOW_MOD_COUNT) { > struct rte_flow_query_count *qc =3D data; > uint64_t counters[2] =3D {0, 0}; > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + int err =3D mlx5_glue->query_counter_set( > + flow->counter->cs, > + counters, > + RTE_DIM(counters), > + IBV_READ_COUNTERS_ATTR_PREFER_CACHED); > +#else > struct ibv_query_counter_set_attr query_cs_attr =3D { > .cs =3D flow->counter->cs, > .query_flags =3D IBV_COUNTER_SET_FORCE_UPDATE, > @@ -3383,7 +3426,7 @@ struct rte_flow * > }; > int err =3D mlx5_glue->query_counter_set(&query_cs_attr, > &query_out); > - > +#endif > if (err) > return rte_flow_error_set > (error, err, > diff --git a/drivers/net/mlx5/mlx5_glue.c b/drivers/net/mlx5/mlx5_glue.c > index 84f9492..b2b39ad 100644 > --- a/drivers/net/mlx5/mlx5_glue.c > +++ b/drivers/net/mlx5/mlx5_glue.c > @@ -211,6 +211,39 @@ > return ibv_dereg_mr(mr); > } >=20 > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > +static struct ibv_counters * > +mlx5_glue_create_counters(struct ibv_context *context, > + struct ibv_counters_init_attr *init_attr) > +{ > + return ibv_create_counters(context, init_attr); > +} > + > +static int > +mlx5_glue_destroy_counters(struct ibv_counters *counters) > +{ > + return ibv_destroy_counters(counters); > +} > + > +static int > +mlx5_glue_attach_counters(struct ibv_counters *counters, > + struct ibv_counter_attach_attr *attr, > + struct ibv_flow *flow) > +{ > + return ibv_attach_counters_point_flow(counters, attr, flow); > +} > + > +static int > +mlx5_glue_query_counters(struct ibv_counters *counters, > + uint64_t *counters_value, > + uint32_t ncounters, > + uint32_t flags) > +{ > + return ibv_read_counters(counters, counters_value, ncounters, flags); > +} > +#endif > + > +#ifndef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > static struct ibv_counter_set * > mlx5_glue_create_counter_set(struct ibv_context *context, > struct ibv_counter_set_init_attr *init_attr) > @@ -262,6 +295,7 @@ > return ibv_query_counter_set(query_attr, cs_data); > #endif > } > +#endif >=20 > static void > mlx5_glue_ack_async_event(struct ibv_async_event *event) > @@ -378,10 +412,17 @@ > .modify_qp =3D mlx5_glue_modify_qp, > .reg_mr =3D mlx5_glue_reg_mr, > .dereg_mr =3D mlx5_glue_dereg_mr, > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + .create_counter_set =3D mlx5_glue_create_counters, > + .destroy_counter_set =3D mlx5_glue_destroy_counters, > + .attach_counter_set =3D mlx5_glue_attach_counters, > + .query_counter_set =3D mlx5_glue_query_counters, > +#else > .create_counter_set =3D mlx5_glue_create_counter_set, > .destroy_counter_set =3D mlx5_glue_destroy_counter_set, > .describe_counter_set =3D mlx5_glue_describe_counter_set, > .query_counter_set =3D mlx5_glue_query_counter_set, > +#endif > .ack_async_event =3D mlx5_glue_ack_async_event, > .get_async_event =3D mlx5_glue_get_async_event, > .port_state_str =3D mlx5_glue_port_state_str, > diff --git a/drivers/net/mlx5/mlx5_glue.h b/drivers/net/mlx5/mlx5_glue.h > index e584d36..c5ff3cb 100644 > --- a/drivers/net/mlx5/mlx5_glue.h > +++ b/drivers/net/mlx5/mlx5_glue.h > @@ -89,6 +89,21 @@ struct mlx5_glue { > struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, > size_t length, int access); > int (*dereg_mr)(struct ibv_mr *mr); > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + struct ibv_counters *(*create_counter_set) > + (struct ibv_context *context, > + struct ibv_counters_init_attr *init_attr); > + int (*destroy_counter_set)(struct ibv_counters *cs); > + int (*attach_counter_set) > + (struct ibv_counters *cs, > + struct ibv_counter_attach_attr *attr, > + struct ibv_flow *flow); > + int (*query_counter_set) > + (struct ibv_counters *cs, > + uint64_t *counters_value, > + uint32_t ncounters, > + uint32_t flags); > +#else > struct ibv_counter_set *(*create_counter_set) > (struct ibv_context *context, > struct ibv_counter_set_init_attr *init_attr); > @@ -99,6 +114,7 @@ struct mlx5_glue { > struct ibv_counter_set_description *cs_desc); > int (*query_counter_set)(struct ibv_query_counter_set_attr *query_attr, > struct ibv_counter_set_data *cs_data); > +#endif > void (*ack_async_event)(struct ibv_async_event *event); > int (*get_async_event)(struct ibv_context *context, > struct ibv_async_event *event); > --=20 > 1.8.3.1 >=20