From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-eopbgr80088.outbound.protection.outlook.com [40.107.8.88]) by dpdk.org (Postfix) with ESMTP id D6B121B394 for ; Thu, 4 Oct 2018 01:48:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8SWTkJssNXPcl3b4oIyfZ6/xvTUIp0QjptO56ei5TMo=; b=TFYu+sjXYc/hwaV75r6lLF4hm4vsKL5wKxFys65ST6cUex0WvffWUXQYhyMjdSqW7ltS+qhzNV2q+qW+UuyImiN+s+rdk6fJ8fP/x/qDbrAiSK97uQrTH5yz9H+YifWLQpgjwp9y5thTzlv9rLngDRv8eEz7X/szNga4bVX5K4g= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4011.eurprd05.prod.outlook.com (52.134.68.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1185.23; Wed, 3 Oct 2018 23:48:18 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::1cb0:661b:ecab:6045]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::1cb0:661b:ecab:6045%2]) with mapi id 15.20.1185.024; Wed, 3 Oct 2018 23:48:18 +0000 From: Yongseok Koh To: Slava Ovsiienko CC: "dev@dpdk.org" , Shahaf Shuler Thread-Topic: [dpdk-dev] [PATCH] net/mlx5: flow counters support on the Linux-rdma v19 base Thread-Index: AQHUW3OPOna93CUI60m8UrJq2EU5eA== Date: Wed, 3 Oct 2018 23:48:18 +0000 Message-ID: <20181003234810.GH26206@mtidpdk.mti.labs.mlnx> References: <1534772079-6514-1-git-send-email-viacheslavo@mellanox.com> <1538580540-2224-1-git-send-email-viacheslavo@mellanox.com> In-Reply-To: <1538580540-2224-1-git-send-email-viacheslavo@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: DM5PR21CA0048.namprd21.prod.outlook.com (2603:10b6:3:ed::34) To DB3PR0502MB3980.eurprd05.prod.outlook.com (2603:10a6:8:10::27) authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB3PR0502MB4011; 6:CrDJ2CtMlDELdSHFwawQJmaGFVCdctOqx7u7q+DY6KyFXoXdtbWN6BgEE/Owgf17VrzYl4RU3u3bhkO9Nym90+yJHhTw71cVo97y6dk5ifV1a6KHkl1oEfE/TRi/GSGA+cuVXTruIogSUv/MIL0L99jjYWjVnz7Dloo3bmGeUSXsWNVZxOevZFRncsKUY+oQ3oYW5Qkqaij1N5WpUVDNBJ7qQSp6PL1qxUdXwNWBTM84ns2UlPr7BzW/T5rs6NdY7qHES6roGEuXyTEvHUPMSWKT8kNTerEm4FbYOqDRLDUlEcSkJ74OHfNpfj8Pn2ZGD1zelSFbjDeeiPGSgvtOjdnklwbmKROrxYQT8I9DSaI97SFLaSYR7FN1k/dh+8DG0oJqT8p3HP4vohdEL8R3BsdBNfm/sn0KTqF6gqL+ost5Ezd+t4zWjHtF5vGgviSVfbejxwm1N4XJ9AfDu8baEw==; 5:9a/iDWHDhnCldKx6B16BpsEx7iIQmAKh/y7D1AGmBXNZZ1cA46p54GGP1kr0eZo59Ze7xrlMF6hy2M2hnrhhu+1v21tSFOmytL+1eCOzqgU1LdkVdiaIQ3XA1OAvOCAq8nUVHEv9nTKOnB5nqTsS3cbnwvxNNDy1cBnq4mXkrck=; 7:wM3fv721epGqnyi2uu+PGXTLAvaIXBzN5GZg+y3m6YFOveUN0Pa+uZ/XqD2KO53h0M0kbjX4Fdi2Pk6FwwHqd9rBA0SlFkvFXRS20jL6G0gan8+N6DKwlHSDmwe2FFpOTGnh1nvcoaVNlEOqj17CE+bjIE1CbU0exOtuiPF/0aQ7yR46sQLxiwghs5duRQaI+6ndLZPGybKu44Fc+7IPyfLAa6dRiJ2S2kT1CCcHyrWMuvPLBIEfJTuD9SnYrhoO x-ms-office365-filtering-correlation-id: 6319b569-5c5f-4bca-d1ba-08d6298ab161 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB4011; x-ms-traffictypediagnostic: DB3PR0502MB4011: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3231355)(944501410)(52105095)(10201501046)(3002001)(93006095)(93001095)(6055026)(149066)(150057)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123562045)(20161123564045)(20161123558120)(201708071742011)(7699051); SRVR:DB3PR0502MB4011; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB4011; x-forefront-prvs: 0814A2C7A3 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(366004)(376002)(396003)(136003)(39860400002)(199004)(189003)(256004)(99286004)(2900100001)(52116002)(4744004)(107886003)(102836004)(5660300001)(14444005)(97736004)(33896004)(76176011)(6636002)(446003)(486006)(2906002)(476003)(86362001)(6486002)(5024004)(6506007)(386003)(305945005)(5250100002)(229853002)(7736002)(54906003)(316002)(186003)(105586002)(53936002)(106356001)(9686003)(11346002)(66066001)(6436002)(6512007)(6862004)(68736007)(3846002)(6246003)(1076002)(25786009)(71200400001)(71190400001)(4326008)(478600001)(81166006)(14454004)(81156014)(8936002)(8676002)(6116002)(33656002)(26005)(309714004); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4011; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: NYu3OlQ5UsbI2mMn2++C6VvUEvKflAfHl9CQKbktVlcnF69HzuBMFIhl5QvBDiFLnmMSsiAbytHTIdPmqCTKCehHugi9dmfK34o2Xd27KSjEtTaLqW16uPo3ExKggolZX4gjSLrVfmmEimWSNRrOyxPZBPuJO46HKRKxVqfBzc91aydPjZx4rNpv0f0s6Q6KmXb7GrihBpoQtcRcTQSP6wAapXXQRK8DdU4LH9DaRCf1T30lGFaJYsfVEBnZBblouglDteLt3q0zfOgO2BjFlzu+HZp+oD22HDeFUk6RRgLtzi6GJwzmqol8dvR8LNxsrRZmRL2wJpOmbQLUjawUGiKrdXMomwWKEdvcb2hd/co= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6319b569-5c5f-4bca-d1ba-08d6298ab161 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Oct 2018 23:48:18.2190 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4011 Subject: Re: [dpdk-dev] [PATCH] net/mlx5: flow counters support on the Linux-rdma v19 base X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 23:48:21 -0000 On Wed, Oct 03, 2018 at 03:29:12PM +0000, Slava Ovsiienko wrote: > Mellanox mlx5 PMD supports Flow Counters via Verbs library. > The current implementation is based on the Mellanox proprietary > Verbs library included in MLNX OFED packages. The Flow Counter > support is recently added into linux-rdma release (v19), > so the mlx5 PMD update is needed to provide Counter feature > on the base of linux-rdma. >=20 > mlx5 PMD can be compiled with MLNX OFED or linux-rdma v19+ > and provide flow counters for both. >=20 > Signed-off-by: Viacheslav Ovsiienko > --- > drivers/net/mlx5/Makefile | 10 ++++++++ > drivers/net/mlx5/mlx5.c | 6 +++++ > drivers/net/mlx5/mlx5_flow.c | 9 ++++++- > drivers/net/mlx5/mlx5_flow.h | 4 +++ > drivers/net/mlx5/mlx5_flow_verbs.c | 52 ++++++++++++++++++++++++++++++--= ------ > drivers/net/mlx5/mlx5_glue.c | 41 ++++++++++++++++++++++++++++++ > drivers/net/mlx5/mlx5_glue.h | 16 ++++++++++++ > 7 files changed, 127 insertions(+), 11 deletions(-) >=20 > diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile > index ca1de9f..e3d2156 100644 > --- a/drivers/net/mlx5/Makefile > +++ b/drivers/net/mlx5/Makefile > @@ -162,6 +162,16 @@ mlx5_autoconf.h.new: $(RTE_SDK)/buildtools/auto-conf= ig-h.sh > type 'struct ibv_counter_set_init_attr' \ > $(AUTOCONF_OUTPUT) > $Q sh -- '$<' '$@' \ > + HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT \ > + infiniband/verbs.h \ > + type 'struct ibv_counters_init_attr' \ > + $(AUTOCONF_OUTPUT) > + $Q sh -- '$<' '$@' \ > + HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 \ > + infiniband/verbs.h \ > + type 'struct ibv_counters_init_attr' \ > + $(AUTOCONF_OUTPUT) > + $Q sh -- '$<' '$@' \ I still don't understand what is different between the two. These are exact= ly same checking, then why do you need to have two different macros? From this script, HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT is same as HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45, isn't it? And if you make some changes in Makefile, you should also make correspondin= g changes for the meson build. > HAVE_RDMA_NL_NLDEV \ > rdma/rdma_netlink.h \ > enum RDMA_NL_NLDEV \ > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c > index 4be6a1c..81f6ba1 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -739,8 +739,10 @@ > unsigned int mprq_min_stride_num_n =3D 0; > unsigned int mprq_max_stride_num_n =3D 0; > #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT > +#ifndef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > struct ibv_counter_set_description cs_desc =3D { .counter_type =3D 0 }; > #endif > +#endif > struct ether_addr mac; > char name[RTE_ETH_NAME_MAX_LEN]; > int own_domain_id =3D 0; > @@ -1009,11 +1011,15 @@ > DRV_LOG(DEBUG, "checksum offloading is %ssupported", > (config.hw_csum ? "" : "not ")); > #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT > +#ifndef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > config.flow_counter_en =3D !!attr.max_counter_sets; > mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); > DRV_LOG(DEBUG, "counter type =3D %d, num of cs =3D %ld, attributes =3D = %d", > cs_desc.counter_type, cs_desc.num_of_cs, > cs_desc.attributes); > +#else > + config.flow_counter_en =3D 1; > +#endif > #endif > config.ind_table_max_size =3D > attr.rss_caps.max_rwq_indirection_table_size; > diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c > index 8007bf1..652580c 100644 > --- a/drivers/net/mlx5/mlx5_flow.c > +++ b/drivers/net/mlx5/mlx5_flow.c > @@ -2306,6 +2306,13 @@ struct rte_flow * > if (flow->actions & MLX5_ACTION_COUNT) { > struct rte_flow_query_count *qc =3D data; > uint64_t counters[2] =3D {0, 0}; > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + int err =3D mlx5_glue->query_counter_set( > + flow->counter->cs, > + counters, > + RTE_DIM(counters), > + IBV_READ_COUNTERS_ATTR_PREFER_CACHED); > +#else > struct ibv_query_counter_set_attr query_cs_attr =3D { > .cs =3D flow->counter->cs, > .query_flags =3D IBV_COUNTER_SET_FORCE_UPDATE, > @@ -2316,7 +2323,7 @@ struct rte_flow * > }; > int err =3D mlx5_glue->query_counter_set(&query_cs_attr, > &query_out); > - > +#endif > if (err) > return rte_flow_error_set > (error, err, > diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h > index 10d700a..a3b82dd 100644 > --- a/drivers/net/mlx5/mlx5_flow.h > +++ b/drivers/net/mlx5/mlx5_flow.h > @@ -222,7 +222,11 @@ struct mlx5_flow_counter { > uint32_t shared:1; /**< Share counter ID with other flow rules. */ > uint32_t ref_cnt:31; /**< Reference counter. */ > uint32_t id; /**< Counter ID. */ > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + struct ibv_counters *cs; /**< Holds the counters for the rule. */ > +#else > struct ibv_counter_set *cs; /**< Holds the counters for the rule. */ > +#endif > uint64_t hits; /**< Number of packets matched by the rule. */ > uint64_t bytes; /**< Number of bytes matched by the rule. */ > }; > diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c b/drivers/net/mlx5/mlx5_f= low_verbs.c > index 05ab5fd..1c8bdba 100644 > --- a/drivers/net/mlx5/mlx5_flow_verbs.c > +++ b/drivers/net/mlx5/mlx5_flow_verbs.c > @@ -48,27 +48,32 @@ > static struct mlx5_flow_counter * > flow_verbs_counter_new(struct rte_eth_dev *dev, uint32_t shared, uint32_= t id) > { > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT > struct priv *priv =3D dev->data->dev_private; > struct mlx5_flow_counter *cnt; > =20 > - LIST_FOREACH(cnt, &priv->flow_counters, next) { > - if (!cnt->shared || cnt->shared !=3D shared) > - continue; > - if (cnt->id !=3D id) > - continue; > - cnt->ref_cnt++; > - return cnt; > + if (shared) { > + LIST_FOREACH(cnt, &priv->flow_counters, next) > + if (cnt->shared && cnt->id =3D=3D id) { > + cnt->ref_cnt++; > + return cnt; > + } > } > -#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT > =20 > struct mlx5_flow_counter tmpl =3D { > .shared =3D shared, > .id =3D id, > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + .cs =3D mlx5_glue->create_counter_set > + (priv->ctx, > + &(struct ibv_counters_init_attr){0}), > +#else > .cs =3D mlx5_glue->create_counter_set > (priv->ctx, > &(struct ibv_counter_set_init_attr){ > .counter_set_id =3D id, > }), > +#endif > .hits =3D 0, > .bytes =3D 0, > }; > @@ -77,17 +82,40 @@ > rte_errno =3D errno; > return NULL; > } > + > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + struct ibv_counter_attach_attr attach_attr =3D {0}; > + int ret; > + > + attach_attr.counter_desc =3D IBV_COUNTER_PACKETS; > + attach_attr.index =3D 0; > + ret =3D ibv_attach_counters_point_flow(tmpl.cs, &attach_attr, NULL); > + if (!ret) { > + attach_attr.counter_desc =3D IBV_COUNTER_BYTES; > + attach_attr.index =3D 1; > + ret =3D ibv_attach_counters_point_flow(tmpl.cs, > + &attach_attr, > + NULL); > + } > + if (ret) { > + claim_zero(mlx5_glue->destroy_counter_set(tmpl.cs)); > + rte_errno =3D ret; > + return NULL; > + } > +#endif > cnt =3D rte_calloc(__func__, 1, sizeof(*cnt), 0); > if (!cnt) { > + claim_zero(mlx5_glue->destroy_counter_set(tmpl.cs)); > rte_errno =3D ENOMEM; > return NULL; > } > *cnt =3D tmpl; > LIST_INSERT_HEAD(&priv->flow_counters, cnt, next); > return cnt; > -#endif > +#else > rte_errno =3D ENOTSUP; > return NULL; > +#endif > } > =20 > /** > @@ -947,7 +975,7 @@ > flow->counter =3D flow_verbs_counter_new(dev, count->shared, > count->id); > if (!flow->counter) > - return rte_flow_error_set(error, ENOTSUP, > + return rte_flow_error_set(error, rte_errno, > RTE_FLOW_ERROR_TYPE_ACTION, > action, > "cannot get counter" > @@ -955,7 +983,11 @@ > } > *action_flags |=3D MLX5_ACTION_COUNT; > #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + counter.counters =3D flow->counter->cs; > +#else > counter.counter_set_handle =3D flow->counter->cs->handle; > +#endif > flow_verbs_spec_add(dev_flow, &counter, size); > #endif > return 0; > diff --git a/drivers/net/mlx5/mlx5_glue.c b/drivers/net/mlx5/mlx5_glue.c > index 48590df..785234c 100644 > --- a/drivers/net/mlx5/mlx5_glue.c > +++ b/drivers/net/mlx5/mlx5_glue.c > @@ -211,6 +211,39 @@ > return ibv_dereg_mr(mr); > } > =20 > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > +static struct ibv_counters * > +mlx5_glue_create_counters(struct ibv_context *context, > + struct ibv_counters_init_attr *init_attr) > +{ > + return ibv_create_counters(context, init_attr); > +} > + > +static int > +mlx5_glue_destroy_counters(struct ibv_counters *counters) > +{ > + return ibv_destroy_counters(counters); > +} > + > +static int > +mlx5_glue_attach_counters(struct ibv_counters *counters, > + struct ibv_counter_attach_attr *attr, > + struct ibv_flow *flow) > +{ > + return ibv_attach_counters_point_flow(counters, attr, flow); > +} > + > +static int > +mlx5_glue_query_counters(struct ibv_counters *counters, > + uint64_t *counters_value, > + uint32_t ncounters, > + uint32_t flags) > +{ > + return ibv_read_counters(counters, counters_value, ncounters, flags); > +} > +#endif > + > +#ifndef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > static struct ibv_counter_set * > mlx5_glue_create_counter_set(struct ibv_context *context, > struct ibv_counter_set_init_attr *init_attr) > @@ -262,6 +295,7 @@ > return ibv_query_counter_set(query_attr, cs_data); > #endif > } > +#endif > =20 > static void > mlx5_glue_ack_async_event(struct ibv_async_event *event) > @@ -420,10 +454,17 @@ > .modify_qp =3D mlx5_glue_modify_qp, > .reg_mr =3D mlx5_glue_reg_mr, > .dereg_mr =3D mlx5_glue_dereg_mr, > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + .create_counter_set =3D mlx5_glue_create_counters, > + .destroy_counter_set =3D mlx5_glue_destroy_counters, > + .attach_counter_set =3D mlx5_glue_attach_counters, > + .query_counter_set =3D mlx5_glue_query_counters, > +#else > .create_counter_set =3D mlx5_glue_create_counter_set, > .destroy_counter_set =3D mlx5_glue_destroy_counter_set, > .describe_counter_set =3D mlx5_glue_describe_counter_set, > .query_counter_set =3D mlx5_glue_query_counter_set, > +#endif > .ack_async_event =3D mlx5_glue_ack_async_event, > .get_async_event =3D mlx5_glue_get_async_event, > .port_state_str =3D mlx5_glue_port_state_str, > diff --git a/drivers/net/mlx5/mlx5_glue.h b/drivers/net/mlx5/mlx5_glue.h > index f6e4e38..504d487 100644 > --- a/drivers/net/mlx5/mlx5_glue.h > +++ b/drivers/net/mlx5/mlx5_glue.h > @@ -96,6 +96,21 @@ struct mlx5_glue { > struct ibv_mr *(*reg_mr)(struct ibv_pd *pd, void *addr, > size_t length, int access); > int (*dereg_mr)(struct ibv_mr *mr); > +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT_V45 > + struct ibv_counters *(*create_counter_set) > + (struct ibv_context *context, > + struct ibv_counters_init_attr *init_attr); > + int (*destroy_counter_set)(struct ibv_counters *cs); > + int (*attach_counter_set) > + (struct ibv_counters *cs, > + struct ibv_counter_attach_attr *attr, > + struct ibv_flow *flow); > + int (*query_counter_set) > + (struct ibv_counters *cs, > + uint64_t *counters_value, > + uint32_t ncounters, > + uint32_t flags); > +#else > struct ibv_counter_set *(*create_counter_set) > (struct ibv_context *context, > struct ibv_counter_set_init_attr *init_attr); > @@ -106,6 +121,7 @@ struct mlx5_glue { > struct ibv_counter_set_description *cs_desc); > int (*query_counter_set)(struct ibv_query_counter_set_attr *query_attr, > struct ibv_counter_set_data *cs_data); > +#endif > void (*ack_async_event)(struct ibv_async_event *event); > int (*get_async_event)(struct ibv_context *context, > struct ibv_async_event *event); > --=20 > 1.8.3.1 >=20