From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A045DA04FD; Mon, 23 May 2022 06:07:47 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7B05540156; Mon, 23 May 2022 06:07:47 +0200 (CEST) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id 066A94014F for ; Mon, 23 May 2022 06:07:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653278865; x=1684814865; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=EnL9ipUiQZPElkLKobo+ZVZTM7Q793L4jxNo0OAlEfQ=; b=bAVNUuhgrcca+opwi4UNy32KgK1OGQXa6MDI2JtwPrBbqhpds1WW9Hd8 Ihtq0GKsjGGxqnUyZfI9bhNfvk8Tnf/ZOd2uN+IRdzy/im6m6D7qyEh0R wNtZF3xU5AXJqzpR+s5woBPOnkfsG9pgmR59QEASjZ3b2jlbaK/VBn5oA jVlGL9MlhH9Kaxhpvit6TPjesNu7h7PTE0piwvVhEoFRsL/8zKFkEMUmX JphTGr/6ka2A/ExqilgrdVCYj57h5FPKJzm66ib4w/8ugbDGufiNZCqWm CaBvFxDA3TT0jydVCG2av54H+wcIkk8gxKTKLB5kARAEGK1NR/P7zPo0V A==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="298407358" X-IronPort-AV: E=Sophos;i="5.91,245,1647327600"; d="scan'208";a="298407358" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 May 2022 21:07:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,245,1647327600"; d="scan'208";a="629161984" Received: from fmsmsx605.amr.corp.intel.com ([10.18.126.85]) by fmsmga008.fm.intel.com with ESMTP; 22 May 2022 21:07:44 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx605.amr.corp.intel.com (10.18.126.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Sun, 22 May 2022 21:07:43 -0700 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Sun, 22 May 2022 21:07:43 -0700 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.100) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Sun, 22 May 2022 21:07:43 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X0gRw8CqKOMbOgUxAapj9TPMJOseQ/LHSU5Sma+b9hn1c5ABpWrOjlEPXr+5cTv5fIsg09+tEqilWR8atLYb6OGghyJq7KNeFSzpWapC5qEiOg3xMGZTxETWVHZ5GmnIqvHfxIa02olACq+6sw0rdhbnu54LMz3ZlDf+0zZj8DoKp83Zfl0OslJD2IVqIx3QhWF3fiFgQ/G44A8X2kPl7AqhGZaPpispCtN0ql0g6oLalxjKhcT1dQ8BIp1jwXwe20vnKrE2IjSEeQ2W0SFgMx89rRVO3Y3KvS37TYbiGELoCUeWx7j5I88C97f7maTodOP+awjAQ3+Z+rccSuV7IQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tgpEuBiKLFRNaDfAzPEy02TpNYQlSyvM6OWgJ1d33Mc=; b=hoaHkq0IB3zDDIkCV4joHNHcxE2piHvD+dS4c7uhm/OB+xG0+55rOWkEQbblDhyAozAW7pUBWcnq8/BHeBFYZpIWU8MsHcOh9cIQu4AvasYnkxetdlRZtBHFpijwm6GTvSqZRj+kGL/V0EVe4uhBuCeR6s9mA18ErEhFlMhldCPrAWHfZUHXw3dEVSQnhRFa2NKfzfk3FAx5ZmpITphEjKbMq48YxIwcHTHxzmINau2x7Dp1zbJRXJ5Z78bun4W+L1oZN2/JXnJAFyYipylcCSfVdg/P47uDTRRfL78mNJxLFbWKdXS6ZTdYap5qWrvgFdusNyeihAueRAeZBr24kg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SN6PR11MB3504.namprd11.prod.outlook.com (2603:10b6:805:d0::17) by CY4PR11MB1639.namprd11.prod.outlook.com (2603:10b6:910:f::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.17; Mon, 23 May 2022 04:07:41 +0000 Received: from SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::54e1:e802:17c8:93f]) by SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::54e1:e802:17c8:93f%7]) with mapi id 15.20.5273.022; Mon, 23 May 2022 04:07:41 +0000 From: "Xia, Chenbo" To: "Pei, Andy" , "dev@dpdk.org" CC: "maxime.coquelin@redhat.com" , "Cao, Gang" , "Liu, Changpeng" , "Xu, Rosen" , "Xiao, QimaiX" Subject: RE: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device Thread-Topic: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device Thread-Index: AQHYardUR+EHgr2hvkqPhtZ2Emet4K0r3hpw Date: Mon, 23 May 2022 04:07:41 +0000 Message-ID: References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> <1652876035-70513-5-git-send-email-andy.pei@intel.com> In-Reply-To: <1652876035-70513-5-git-send-email-andy.pei@intel.com> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 632461e1-39a2-4370-861a-08da3c71c823 x-ms-traffictypediagnostic: CY4PR11MB1639:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: uqVfdAfhmDxoD8YaMH7nv/3VSwZpoFgBzA7X9XLkBeviS1o3tdl/h7ZcUPmU3seCU16kcO2psTDdNH/SyBCRdDyzPJVq4bjQA9WkwDoDmLzql7N3c813+WYSfu1Mjvhj2sjNHxn7MV5l1FdFa4gSrMAD5QgxgRFhI8QL6XQ4i5HX8vjFK/W+L4rtHLI6czH54U2IdmTwGD2ResZw7l02OeMWyOZIE8Sd5t2sU3dmK/lZfRxtz/CNqyZkA6ViJUV4S+xcnCfE8wvUV6nyzmqpHeM7iv0M7LHdaDtzsHypRWCgeqkMJSVeXfe69FCxIxW24+wzVg8UMsj/9k6fInchHRO+QrVqIykBnDRgbDQ7sca1t/CiZejNCmKx6Ln2SDe1tv+n88kAj+beEc7pct6vW3wBOHW9mKAD8z6IEvhQRgo32Hp8/+uA5N/nm84KxFbWkDLSTMVzbnOVhDeyEtVLjESW7TzT2im2OP9CVnMbbthjVERbg2sVpuiLp7Dhzpq36GUqrwa/VsIPC4TZaAzQ8Tauu3LHaIJYcQN3h4iq8dpKOKTPEQ6Wrs9/K8POKC7kflvzcJiAs09nBrBlb8stFb6r3o31TJlgOi512470O2etcAwoq3mRifTOkRTQFct5E5H8uy8YvfKSPFhFnCZF3B8mw+B4+hTktYZHsb23CyqECJxCg5lxLehpE5vl3JB/ihKLQmUC1zhTeHGx4ltgnw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR11MB3504.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(5660300002)(55016003)(83380400001)(107886003)(186003)(82960400001)(8936002)(2906002)(38100700002)(38070700005)(52536014)(122000001)(76116006)(54906003)(33656002)(9686003)(7696005)(6506007)(4326008)(86362001)(53546011)(316002)(110136005)(8676002)(66446008)(64756008)(66556008)(508600001)(71200400001)(26005)(66476007)(66946007); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?rAijnf7ZXsnufC2odKQm4l/sNlzINp19GHGyIG3JCqCFtIvF/zMjklcN4xnP?= =?us-ascii?Q?CBct+saw4AB+Z2Eq0WqctLH0KrkWsd+21puV38g9/R04LzIumqlqVZo/FUmH?= =?us-ascii?Q?CskaABgICzKkRpN29oHYRIdxrv9MLlihcuP7unGMaC5RxmgYAhCZZNegoz/g?= =?us-ascii?Q?JAreURfQoKOIuauOZp4V0dEBxgloKKB8GRkeWjiNgzvcVPzhc0yvw1bKDXV+?= =?us-ascii?Q?4xw7CqL/wwtXiBWaC0GVpuiMHfiKwjc6fQnaX2yCoodmBzLfQZNMcn1INu9q?= =?us-ascii?Q?7IJsy+ptiELDDG/ftgCDo/Rp/Gp64j4aPVQRnqvY5+fG0QrJtP0o1fq7yjkN?= =?us-ascii?Q?onH6zlrBPZWNlhWvyuvX7o8idbToeDrpdESo+IG8bkcBMCgpI2eg4Th6N2Jr?= =?us-ascii?Q?sJ02V4ytbVtjvP3Btq2PqKXpX+RmzPbEcMpcM2Z1tgyG980Sii2zMoprsUAn?= =?us-ascii?Q?JmUjYAXvaVOz7ZpRcpGzNU6O0syBe0uiPo7OjJ/1bX3HeamYT16GYiN9IA0s?= =?us-ascii?Q?HM26Gw1rBCoQi0UVefh610wZINguAbIVony74zYGmil5wXJZjW8CzpIOy7DS?= =?us-ascii?Q?sx9QZyQLk77bY7X4Dhak5OqKtm1wdlRaexK24pefsZI3rIZJ76kMSTUl2iZK?= =?us-ascii?Q?OCxO/FrGwsCy7f0Hx9vyFfQXt7/cCNcJBlQTmlfl51+ebjPCzJCy+GfyracE?= =?us-ascii?Q?yXo2cHNdom83r2F5PfWO2PrUyQY2tI5J5ud/T9dp9IV2xXQOWOE1sQ6wPn8j?= =?us-ascii?Q?iqUZrsW2lJtA1YcAiZLQW+iQerXTof+p6jcWXFRM9ieexREA6Tcct+skBLY+?= =?us-ascii?Q?cD9XaEzzs1v5gQhJ8qI4Po/8HPUuhEryFcUW1enISrSebON5JU1QVuinBwAz?= =?us-ascii?Q?YzNJfkf6ZdmgQQkfZHpgbFS3aQxxvP84SZU7A4gO14RfGWEJAo/wzJfEOy5O?= =?us-ascii?Q?jlPkcwzX+3s66FrF94NZyYa6oWfs1QCnWtBhso72sIStaoxGRUTK14VTrW+p?= =?us-ascii?Q?zQdw/eb67Y+EfoKsfVlnDghzbtzgVO2hoCdcidFGvkkJS00Ht6ELMp/HmxY3?= =?us-ascii?Q?Ucm3BAMxq0auUh0794ycxkjvDWCDCx5bp8jKcR6IxB9PY6v2VztGBPFsoCQY?= =?us-ascii?Q?bk8g7qV8IneYqK6ihkFvBMCv6cKty1H184Bv5570sfHJWI0anyokQetIElre?= =?us-ascii?Q?bYbxhIlqmp+uaJi4LE/QrOi8z8HAfFuE4TOodZL/3uyUgyg3ym3yD3AEMFur?= =?us-ascii?Q?ycexYXmnEcqAe2SWO5DzPcl5dU2rzum+RGObtwCRskSvaUJKgOH3MB8RYRF2?= =?us-ascii?Q?ONRSPU5Qgt8eGrUU0uV0zYhfkgVUN4TbRFxCLDOwnQOBkEjzzxbm4BhuDLWc?= =?us-ascii?Q?NFqYsd6xp/hZJq+5asjH/KnHHNrrobUgu2xS7lRh4DhL7O+LJ4sWxBH8F/hd?= =?us-ascii?Q?hlJIWdTu3cXapdw410OwzGRQgehgYdzIBguxaH5FQ6KAXsEsy7xXbc6qkmwm?= =?us-ascii?Q?1/0tUn3FCc2mWOdUYuw7mxT4iWw2EAE/hCzQJZmfdvjn/P7ZNzlxi3YRFqtV?= =?us-ascii?Q?lSlzmPfMxmyLbydiV3pajIgyZUBsgUPrIPXqQ/VjtLx6+xNsIt5KOdbocaRN?= =?us-ascii?Q?MYxcJhFFNUOpY1wovWLN45y58PrDWoxLwddpDnnDu67qlDTa6XnYH7kqXegQ?= =?us-ascii?Q?3VGPo2L+/e/gu42qNSP8lpJz345+6CNwxxaM4AOHdbRgo/FdAtd6ZUNcidcE?= =?us-ascii?Q?QqHazmIFPw=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR11MB3504.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 632461e1-39a2-4370-861a-08da3c71c823 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 May 2022 04:07:41.2932 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 7kFXkYJKqu3+q43q1tN/XqV1gX0XDpcEjlzMeyAJtvk/2+kKjcXOmuJr5KDrXOuGlRBgg5vJLt+6/Hy53MsIvw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR11MB1639 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Pei, Andy > Sent: Wednesday, May 18, 2022 8:14 PM > To: dev@dpdk.org > Cc: Xia, Chenbo ; maxime.coquelin@redhat.com; Cao, > Gang ; Liu, Changpeng ; Xu, > Rosen ; Xiao, QimaiX > Subject: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device >=20 > For virtio blk device, re-use part of ifc driver ops. > Implement ifcvf_blk_get_config for virtio blk device. > Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio > blk device. >=20 > Signed-off-by: Andy Pei > --- > drivers/vdpa/ifc/base/ifcvf.h | 4 ++ > drivers/vdpa/ifc/ifcvf_vdpa.c | 91 > ++++++++++++++++++++++++++++++++++++++++++- > 2 files changed, 94 insertions(+), 1 deletion(-) >=20 > diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.= h > index 483d38b..244de46 100644 > --- a/drivers/vdpa/ifc/base/ifcvf.h > +++ b/drivers/vdpa/ifc/base/ifcvf.h > @@ -67,6 +67,10 @@ > #define IFCVF_32_BIT_MASK 0xffffffff >=20 >=20 > +#ifndef VHOST_USER_PROTOCOL_F_CONFIG > +#define VHOST_USER_PROTOCOL_F_CONFIG 9 > +#endif > + > struct ifcvf_pci_cap { > u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ > u8 cap_next; /* Generic PCI field: next ptr. */ > diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.= c > index be0efd3..350214a 100644 > --- a/drivers/vdpa/ifc/ifcvf_vdpa.c > +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c > @@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info { > 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ > 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ > 1ULL << VHOST_USER_PROTOCOL_F_STATUS) > + > +#define VDPA_BLK_PROTOCOL_FEATURES \ > + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) > + > static int > ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t > *features) > { > @@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info { > return device_id; > } >=20 > +static int > +ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t len) > +{ > + struct virtio_blk_config *dev_cfg; > + struct ifcvf_internal *internal; > + struct rte_vdpa_device *vdev; > + struct internal_list *list; > + uint32_t i; > + uint64_t capacity =3D 0; > + uint8_t *byte; > + > + if (len < sizeof(struct virtio_blk_config)) { > + DRV_LOG(ERR, "Invalid len: %u, required: %u", > + len, (uint32_t)sizeof(struct virtio_blk_config)); > + return -1; > + } I believe it should not be > sizeof(struct virtio_blk_config) too? And one question is should we limit only reading the whole blk config. I guess we are having this check because of current usage of QEMU?=20 Although this is fine as it's vendor-specific logic, just wondering if we should have this limit for blk device. Thanks, Chenbo > + > + vdev =3D rte_vhost_get_vdpa_device(vid); > + list =3D find_internal_resource_by_vdev(vdev); > + if (list =3D=3D NULL) { > + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); > + return -1; > + } > + > + internal =3D list->internal; > + > + for (i =3D 0; i < sizeof(struct virtio_blk_config); i++) > + config[i] =3D *((u8 *)internal->hw.blk_cfg + i); > + > + dev_cfg =3D (struct virtio_blk_config *)internal->hw.blk_cfg; > + > + /* cannot read 64-bit register in one attempt, so read byte by byte. > */ > + for (i =3D 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { > + byte =3D (uint8_t *)&internal->hw.blk_cfg->capacity + i; > + capacity |=3D (uint64_t)*byte << (i * 8); > + } > + /* The capacity is number of sectors in 512-byte. > + * So right shift 1 bit we get in K, > + * another right shift 10 bits we get in M, > + * right shift 10 more bits, we get in G. > + * To show capacity in G, we right shift 21 bits in total. > + */ > + DRV_LOG(DEBUG, "capacity : %"PRIu64"G", capacity >> 21); > + > + DRV_LOG(DEBUG, "size_max : 0x%08x", dev_cfg->size_max); > + DRV_LOG(DEBUG, "seg_max : 0x%08x", dev_cfg->seg_max); > + DRV_LOG(DEBUG, "blk_size : 0x%08x", dev_cfg->blk_size); > + DRV_LOG(DEBUG, "geometry"); > + DRV_LOG(DEBUG, " cylinders: %u", dev_cfg->geometry.cylinders); > + DRV_LOG(DEBUG, " heads : %u", dev_cfg->geometry.heads); > + DRV_LOG(DEBUG, " sectors : %u", dev_cfg->geometry.sectors); > + DRV_LOG(DEBUG, "num_queues: 0x%08x", dev_cfg->num_queues); > + > + DRV_LOG(DEBUG, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n", > + config[0], config[1], config[2], config[3], config[4], > + config[5], config[6], config[7]); > + return 0; > +} > + > +static int > +ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, > + uint64_t *features) > +{ > + RTE_SET_USED(vdev); > + > + *features =3D VDPA_SUPPORTED_PROTOCOL_FEATURES; > + *features |=3D VDPA_BLK_PROTOCOL_FEATURES; > + return 0; > +} > + > +static struct rte_vdpa_dev_ops ifcvf_blk_ops =3D { > + .get_queue_num =3D ifcvf_get_queue_num, > + .get_features =3D ifcvf_get_vdpa_features, > + .set_features =3D ifcvf_set_features, > + .get_protocol_features =3D ifcvf_blk_get_protocol_features, > + .dev_conf =3D ifcvf_dev_config, > + .dev_close =3D ifcvf_dev_close, > + .set_vring_state =3D ifcvf_set_vring_state, > + .migration_done =3D NULL, > + .get_vfio_group_fd =3D ifcvf_get_vfio_group_fd, > + .get_vfio_device_fd =3D ifcvf_get_vfio_device_fd, > + .get_notify_area =3D ifcvf_get_notify_area, > + .get_config =3D ifcvf_blk_get_config, > +}; > + > struct rte_vdpa_dev_info dev_info[] =3D { > { > .features =3D (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | > @@ -1211,7 +1300,7 @@ struct rte_vdpa_dev_info dev_info[] =3D { > { > .features =3D (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | > (1ULL << VHOST_F_LOG_ALL), > - .ops =3D NULL, > + .ops =3D &ifcvf_blk_ops, > }, > }; >=20 > -- > 1.8.3.1