From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 02ABFA04DE; Fri, 23 Oct 2020 05:32:45 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 60A1F677E; Fri, 23 Oct 2020 05:32:44 +0200 (CEST) Received: from alln-iport-2.cisco.com (alln-iport-2.cisco.com [173.37.142.89]) by dpdk.org (Postfix) with ESMTP id 77D1F677D for ; Fri, 23 Oct 2020 05:32:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=8741; q=dns/txt; s=iport; t=1603423962; x=1604633562; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=0wUZ+vFN6sL+Tacuz+X5o/JYpDfRrfn4EVWjIbcKK08=; b=hmt4XAm98nEdh0q+WU2IJ0dlYrF4MIJESuNYr3/aCGhuCdJmK4ZEr8yk JXlfdWwzHT7jWV4v+iPENfS4KV5la5hi5QiiCFLfTsTNFsIlkFJA2a/gx Z9BX9PuTie7ck5PWmsu9NWReCToZrO1LTjV9rh2oEOo5joXLyQmV3qc4i s=; IronPort-PHdr: =?us-ascii?q?9a23=3AY2fA3By7PzMI9NXXCy+N+z0EezQntrPoPwUc9p?= =?us-ascii?q?sgjfdUf7+++4j5ZRaDt+5qiFbTU4HS8f8CgO3T4OjsWm0FtJCGtn1KMJlBTA?= =?us-ascii?q?QMhshemQs8SNWEBkv2IL+PDWQ6Ec1OWUUj8yS9Nk5YS935Y1TIrHm79zNUER?= =?us-ascii?q?L6ZkJ5I+3vEdvUiMK6n+m555zUZVBOgzywKbN/JRm7t0PfrM4T1IBjMa02jB?= =?us-ascii?q?DOpyhF?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0CcCQB2TpJf/4YNJK1gHgEBCxIMQIM?= =?us-ascii?q?hUQeBSS8sCod7A41OmHqBQoERA1ULAQEBDQEBLQIEAQGESgKCCQIlOBMCAwE?= =?us-ascii?q?BCwEBBQEBAQIBBgRthWEMhXIBAQEEEhUTBgEBNwELBAIBCA4DBAEBHxAyHQg?= =?us-ascii?q?CBAENBQgahVADLQEBpmcCgTmIaHSBATODBAEBBYUBGIIQCYE4gnKGNIQTG4F?= =?us-ascii?q?BP4ERQ4JNPoQjHINIgiyQDRqNaJl6CoJqmxuDF4oNlDqTPKA3AgQCBAUCDgE?= =?us-ascii?q?BBYFrIyqBLXAVgyRQFwINjh8LAReDToNrhEKBcAE4dDgCBgoBAQMJfIw7AYE?= =?us-ascii?q?QAQE?= X-IronPort-AV: E=Sophos;i="5.77,404,1596499200"; d="scan'208";a="590615402" Received: from alln-core-12.cisco.com ([173.36.13.134]) by alln-iport-2.cisco.com with ESMTP/TLS/DHE-RSA-SEED-SHA; 23 Oct 2020 03:32:40 +0000 Received: from XCH-ALN-005.cisco.com (xch-aln-005.cisco.com [173.36.7.15]) by alln-core-12.cisco.com (8.15.2/8.15.2) with ESMTPS id 09N3WeiP019271 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=FAIL); Fri, 23 Oct 2020 03:32:40 GMT Received: from xhs-aln-002.cisco.com (173.37.135.119) by XCH-ALN-005.cisco.com (173.36.7.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Oct 2020 22:32:39 -0500 Received: from xhs-rtp-002.cisco.com (64.101.210.229) by xhs-aln-002.cisco.com (173.37.135.119) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Oct 2020 22:32:39 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (64.101.32.56) by xhs-rtp-002.cisco.com (64.101.210.229) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Thu, 22 Oct 2020 23:32:39 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jxSm45E9f9EPl6gsBKxWUPog1XXtKkFYcoPVbqw5HNdKA9ec6EUSDEelPPnjOVsKwqVqAazZJVRfTDiDZOb63oBtGJvXlpAoI0HR/B075bRSdHKYCBig7TwijBwA9mTWCLMKYQtPwzla3WBTwOuHiYTbvekMeSGcSpSuLk//eCVsk3Hg5vdQdEoA5auEl/xFE8MU9DeMN+R623pfhrFWDNkYrmxis0ydX3lOoyjyhJcPu8ScMiIiho5JazliPEKsqhIKFl9P0yYFHKwBhAudGlraIDDLV0k/UhwSoyLmITa8dEBlT2qBJTE75JCu8HUb4vo8WuqrapQC92pm6eVNuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FsbHrYje4l96U1tii7OuVhbR04FBs7PkG6peKoA+wQg=; b=BX+tO4OhLCasCRh656i7HClBKNfUNBDcLBGzv2CY5rhbn8h5V2eBsqpLSE1pgZpkj0ng8bdDiKUc7xJ2SQUrhqqenV9mYw1TuHmo+zkckcj8E9JphvV6jW7czBEkfIWvABjsn+f9D0H057Zof3GphQzHsizB8dwbp3R1PU77QloDf9dzvTOCVcFkF8Cu6mqPerpEyINrowyMAren0Lx47qOdZENZ0nHenZaKsam6DRWq8d96EgqE32mtTVXbK6xCCkXfC6sW1qYKy7DIhnw9mDLaXTQkzGS19L5XeocQe+yJf1veku+MyxT1GoFqTg+AL1lq3Wu2jafXca2ah5ASuA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cisco.com; dmarc=pass action=none header.from=cisco.com; dkim=pass header.d=cisco.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.onmicrosoft.com; s=selector2-cisco-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FsbHrYje4l96U1tii7OuVhbR04FBs7PkG6peKoA+wQg=; b=cNCqau2y/gZRx3S2Bh7uXpkrJqkTBIKutrPwPABIHCCVnWDE4wvq0eSm+1kBQp4EJlnQAGGVekXtqhRQsyluap2dlbRC1kEaXM4GKv3lrKYa8AuoJ0dX28iKmOsq9Vr6rR+rTfnV1Nxj3BTmbZWXTi+HZdqDwKPG8Aw1tgv2fww= Received: from DM5PR11MB1690.namprd11.prod.outlook.com (2603:10b6:3:15::11) by DM5PR11MB2012.namprd11.prod.outlook.com (2603:10b6:3:10::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3499.18; Fri, 23 Oct 2020 03:32:34 +0000 Received: from DM5PR11MB1690.namprd11.prod.outlook.com ([fe80::69a3:b8b6:4020:9e4f]) by DM5PR11MB1690.namprd11.prod.outlook.com ([fe80::69a3:b8b6:4020:9e4f%8]) with mapi id 15.20.3499.018; Fri, 23 Oct 2020 03:32:34 +0000 From: "Hyong Youb Kim (hyonkim)" To: Andrew Rybchenko , Wenzhuo Lu , Beilei Xing , "Bernard Iremonger" , Jeff Guo , "Ray Kinsella" , Neil Horman , Ajit Khaparde , Somnath Kotur , "John Daley (johndale)" , Haiyue Wang , Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko , "Rasesh Mody" , Shahed Shaikh , "Andrew Rybchenko" , Thomas Monjalon , Ferruh Yigit CC: "dev@dpdk.org" Thread-Topic: [PATCH v2 12/14] ethdev: remove legacy FDIR filter type support Thread-Index: AQHWqFfIKNk8JyHwe0uvPb7yPKADjKmkhO+A Date: Fri, 23 Oct 2020 03:32:34 +0000 Message-ID: References: <1603030152-13451-1-git-send-email-arybchenko@solarflare.com> <1603359758-13297-1-git-send-email-arybchenko@solarflare.com> <1603359758-13297-13-git-send-email-arybchenko@solarflare.com> In-Reply-To: <1603359758-13297-13-git-send-email-arybchenko@solarflare.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: solarflare.com; dkim=none (message not signed) header.d=none;solarflare.com; dmarc=none action=none header.from=cisco.com; x-originating-ip: [123.111.163.40] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d6fe748a-625b-4578-bb54-08d8770447ea x-ms-traffictypediagnostic: DM5PR11MB2012: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3383; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: FM0HGHhbO9yH5RG/33gIZXaoAiGk51GWgGxnT3XMqMABKsgG09bS+d53IAnpbNrzO/+zG7fKUz5IjXn4W+2TlQJeCnLnSMOoBQvkVucmA/EzIGLwoZjHWYfYiRT+A61vSkm9EKYI1bRCKdTFjKdfgVi7X4k5/ItpCoteK4WhoC0J5t2MTZNW3uLbf+goJGZaG/zuZWOJDIZwhx2ZG59ljRGI9O8Ss2jhWQHJqd3ddryJwHIeft+KHIvL+O/Rw3edzlrLweG6uFQnsF11SWfJH8fYPANvI5Pzl8SsVqFuKvgzNKxUleHrKjHRLoQgMsCZNRtaz/jesu7NtEhcUH1x1WpbRfpf2BHh5cnSr8lCIT8= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM5PR11MB1690.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(376002)(366004)(136003)(346002)(39860400002)(66446008)(9686003)(8676002)(4326008)(186003)(55016002)(478600001)(7696005)(26005)(6506007)(2906002)(53546011)(8936002)(66946007)(52536014)(66556008)(64756008)(76116006)(7416002)(316002)(86362001)(66476007)(71200400001)(5660300002)(110136005)(83380400001)(33656002)(921003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: yzfFJU1tlBN1Wc/Quv0pQ/eZNxQe9tInlu9iSuqwKEK6kyNPxNe6joq37WZwnxRjumYsXinis8SOXK0JWxHt+jTzsv6uz+lidibq8X+vLvjDplp6+zPW68OGWEJ3nCQsB4IJDB6qScGMndFZTBE7Q1mD3rhRdTzOggqz0L3lVzPNJl6Ko8uYA9KxsL0/qjvOOLr92EnOeX2KuB5pBUbHpt2fj1e77qy8749uId5O/qEFKF6CIOtZcXVMwPwXYl/lYNognAmfJSggzmp/tB5Y1/0RqdJ3mp6iwMLZqfoz5w+1UxsNryDaEuWFjPm/Ai5wylZsNXPQEhesjJWe/c7Ux0fZleXARXuMGT8D2gZ4nNYi4lEtrbXWWF9YQkcm3VFu9vnkWP6V1m6p0RfObGXrPKtp8tmyHgOl0+4/ORmMc745d1DuzK87jeqikD6GgBpuATuYChNLGliytwikbceYwqdGeo2Nk6GQNi0iNd1xyVtrI0kHci/d6+BbOO+TGuZeRDuk5i34zqdE4YtxKWmfqrFdUBQCM5DjBqImVXxJtO1co47K9H7wNRSqGhjVd1qeQR9/UaQqrvgbTrDoiIMPZyyfqdnydNYPOylPSm5AvpXIr9Bxn7lAypXj8L6xIW7RBWMSMrNFXp+noHVPAPi2dA== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM5PR11MB1690.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: d6fe748a-625b-4578-bb54-08d8770447ea X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Oct 2020 03:32:34.2125 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 5ae1af62-9505-4097-a69a-c1553ef7840e X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: jw8/tpeVdxWRxJNmX2IyyUpCJGdq8eX7yyzP9/68soTuJj+X7c7Mflu7n3WsAjGf3m78ktBy+22TxolClvGNlg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR11MB2012 X-OriginatorOrg: cisco.com X-Outbound-SMTP-Client: 173.36.7.15, xch-aln-005.cisco.com X-Outbound-Node: alln-core-12.cisco.com Subject: Re: [dpdk-dev] [PATCH v2 12/14] ethdev: remove legacy FDIR filter type support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Andrew Rybchenko > Sent: Thursday, October 22, 2020 6:43 PM > To: Wenzhuo Lu ; Beilei Xing > ; Bernard Iremonger > ; Jeff Guo ; Ray Kinsella > ; Neil Horman ; Ajit Khaparde > ; Somnath Kotur > ; John Daley (johndale) > ; Hyong Youb Kim (hyonkim) ; > Haiyue Wang ; Matan Azrad ; > Shahaf Shuler ; Viacheslav Ovsiienko > ; Rasesh Mody ; Shahed > Shaikh ; Andrew Rybchenko > ; Thomas Monjalon > ; Ferruh Yigit > Cc: dev@dpdk.org > Subject: [PATCH v2 12/14] ethdev: remove legacy FDIR filter type support >=20 > Instead of FDIR filters RTE flow API should be used. >=20 > Signed-off-by: Andrew Rybchenko > Acked-by: Ajit Khaparde > Acked-by: Haiyue Wang > --- > - cleanup testpmd user guide > - partially cleanup net/i40e documentation, please, review and > suggest how to fix remaining flow_director_filter command [...] > drivers/net/enic/enic.h | 7 - > drivers/net/enic/enic_clsf.c | 168 ---- > drivers/net/enic/enic_ethdev.c | 48 -- [...] > diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h > index 5f0ae395da..079f194275 100644 > --- a/drivers/net/enic/enic.h > +++ b/drivers/net/enic/enic.h > @@ -394,12 +394,6 @@ enic_ring_incr(uint32_t n_descriptors, uint32_t idx) > } >=20 > int dev_is_enic(struct rte_eth_dev *dev); > -void enic_fdir_stats_get(struct enic *enic, > - struct rte_eth_fdir_stats *stats); > -int enic_fdir_add_fltr(struct enic *enic, > - struct rte_eth_fdir_filter *params); > -int enic_fdir_del_fltr(struct enic *enic, > - struct rte_eth_fdir_filter *params); > void enic_free_wq(void *txq); > int enic_alloc_intr_resources(struct enic *enic); > int enic_setup_finish(struct enic *enic); > @@ -464,7 +458,6 @@ bool enic_use_vector_rx_handler(struct rte_eth_dev > *eth_dev); > void enic_pick_rx_handler(struct rte_eth_dev *eth_dev); > void enic_pick_tx_handler(struct rte_eth_dev *eth_dev); > void enic_fdir_info(struct enic *enic); > -void enic_fdir_info_get(struct enic *enic, struct rte_eth_fdir_info *sta= ts); > int enic_vf_representor_init(struct rte_eth_dev *eth_dev, void > *init_params); > int enic_vf_representor_uninit(struct rte_eth_dev *ethdev); > int enic_fm_allocate_switch_domain(struct enic *pf); > diff --git a/drivers/net/enic/enic_clsf.c b/drivers/net/enic/enic_clsf.c > index e206123ba5..1c837a4d09 100644 > --- a/drivers/net/enic/enic_clsf.c > +++ b/drivers/net/enic/enic_clsf.c > @@ -42,17 +42,6 @@ static void copy_fltr_v2(struct filter_v2 *fltr, > const struct rte_eth_fdir_input *input, > const struct rte_eth_fdir_masks *masks); >=20 > -void enic_fdir_stats_get(struct enic *enic, struct rte_eth_fdir_stats *s= tats) > -{ > - *stats =3D enic->fdir.stats; > -} > - > -void enic_fdir_info_get(struct enic *enic, struct rte_eth_fdir_info *inf= o) > -{ > - info->mode =3D (enum rte_fdir_mode)enic->fdir.modes; > - info->flow_types_mask[0] =3D enic->fdir.types_mask; > -} > - > void enic_fdir_info(struct enic *enic) > { > enic->fdir.modes =3D (uint32_t)RTE_FDIR_MODE_PERFECT; > @@ -305,163 +294,6 @@ copy_fltr_v2(struct filter_v2 *fltr, const struct > rte_eth_fdir_input *input, > } > } >=20 > -int enic_fdir_del_fltr(struct enic *enic, struct rte_eth_fdir_filter *pa= rams) > -{ > - int32_t pos; > - struct enic_fdir_node *key; > - /* See if the key is in the table */ > - pos =3D rte_hash_del_key(enic->fdir.hash, params); > - switch (pos) { > - case -EINVAL: > - case -ENOENT: > - enic->fdir.stats.f_remove++; > - return -EINVAL; > - default: > - /* The entry is present in the table */ > - key =3D enic->fdir.nodes[pos]; > - > - /* Delete the filter */ > - vnic_dev_classifier(enic->vdev, CLSF_DEL, > - &key->fltr_id, NULL, NULL); > - rte_free(key); > - enic->fdir.nodes[pos] =3D NULL; > - enic->fdir.stats.free++; > - enic->fdir.stats.remove++; > - break; > - } > - return 0; > -} > - > -int enic_fdir_add_fltr(struct enic *enic, struct rte_eth_fdir_filter *pa= rams) > -{ > - struct enic_fdir_node *key; > - struct filter_v2 fltr; > - int32_t pos; > - uint8_t do_free =3D 0; > - uint16_t old_fltr_id =3D 0; > - uint32_t flowtype_supported; > - uint16_t flex_bytes; > - uint16_t queue; > - struct filter_action_v2 action; > - > - memset(&fltr, 0, sizeof(fltr)); > - memset(&action, 0, sizeof(action)); > - flowtype_supported =3D enic->fdir.types_mask > - & (1 << params->input.flow_type); > - > - flex_bytes =3D ((params->input.flow_ext.flexbytes[1] << 8 & 0xFF00) | > - (params->input.flow_ext.flexbytes[0] & 0xFF)); > - > - if (!enic->fdir.hash || > - (params->input.flow_ext.vlan_tci & 0xFFF) || > - !flowtype_supported || flex_bytes || > - params->action.behavior /* drop */) { > - enic->fdir.stats.f_add++; > - return -ENOTSUP; > - } > - > - /* Get the enicpmd RQ from the DPDK Rx queue */ > - queue =3D enic_rte_rq_idx_to_sop_idx(params->action.rx_queue); > - > - if (!enic->rq[queue].in_use) > - return -EINVAL; > - > - /* See if the key is already there in the table */ > - pos =3D rte_hash_del_key(enic->fdir.hash, params); > - switch (pos) { > - case -EINVAL: > - enic->fdir.stats.f_add++; > - return -EINVAL; > - case -ENOENT: > - /* Add a new classifier entry */ > - if (!enic->fdir.stats.free) { > - enic->fdir.stats.f_add++; > - return -ENOSPC; > - } > - key =3D rte_zmalloc("enic_fdir_node", > - sizeof(struct enic_fdir_node), 0); > - if (!key) { > - enic->fdir.stats.f_add++; > - return -ENOMEM; > - } > - break; > - default: > - /* The entry is already present in the table. > - * Check if there is a change in queue > - */ > - key =3D enic->fdir.nodes[pos]; > - enic->fdir.nodes[pos] =3D NULL; > - if (unlikely(key->rq_index =3D=3D queue)) { > - /* Nothing to be done */ > - enic->fdir.stats.f_add++; > - pos =3D rte_hash_add_key(enic->fdir.hash, params); > - if (pos < 0) { > - dev_err(enic, "Add hash key failed\n"); > - return pos; > - } > - enic->fdir.nodes[pos] =3D key; > - dev_warning(enic, > - "FDIR rule is already present\n"); > - return 0; > - } > - > - if (likely(enic->fdir.stats.free)) { > - /* Add the filter and then delete the old one. > - * This is to avoid packets from going into the > - * default queue during the window between > - * delete and add > - */ > - do_free =3D 1; > - old_fltr_id =3D key->fltr_id; > - } else { > - /* No free slots in the classifier. > - * Delete the filter and add the modified one later > - */ > - vnic_dev_classifier(enic->vdev, CLSF_DEL, > - &key->fltr_id, NULL, NULL); > - enic->fdir.stats.free++; > - } > - > - break; > - } > - > - key->filter =3D *params; > - key->rq_index =3D queue; > - > - enic->fdir.copy_fltr_fn(&fltr, ¶ms->input, > - &enic->rte_dev->data- > >dev_conf.fdir_conf.mask); > - action.type =3D FILTER_ACTION_RQ_STEERING; > - action.rq_idx =3D queue; > - > - if (!vnic_dev_classifier(enic->vdev, CLSF_ADD, &queue, &fltr, > - &action)) { > - key->fltr_id =3D queue; > - } else { > - dev_err(enic, "Add classifier entry failed\n"); > - enic->fdir.stats.f_add++; > - rte_free(key); > - return -1; > - } > - > - if (do_free) > - vnic_dev_classifier(enic->vdev, CLSF_DEL, &old_fltr_id, NULL, > - NULL); > - else{ > - enic->fdir.stats.free--; > - enic->fdir.stats.add++; > - } > - > - pos =3D rte_hash_add_key(enic->fdir.hash, params); > - if (pos < 0) { > - enic->fdir.stats.f_add++; > - dev_err(enic, "Add hash key failed\n"); > - return pos; > - } > - > - enic->fdir.nodes[pos] =3D key; > - return 0; > -} > - > void enic_clsf_destroy(struct enic *enic) > { > uint32_t index; Hi, For enic: Acked-by: Hyong Youb Kim Looks like you mechanically removed the things that caused compilation errors. That is fine. Once this patch series lands, I will post a follow-up cleanup patch for enic: remove the entire enic_clsf.c and remaining traces of fdir in enic.h. Thanks. -Hyong