From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0076.outbound.protection.outlook.com [104.47.2.76]) by dpdk.org (Postfix) with ESMTP id 538D39171; Tue, 1 Aug 2017 12:12:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=9/AaERjBEdOKhviUclw7q4OjqDE0QyxtrojUjx20ElY=; b=wxGTsnTjJSJt8cbjRzCt5FVDexRXkf1kWDruB1FC8Kuu+VeNUepfiEmbzwd9rDtcf70zVLDRHTob4Zg/5OFESl5lCC9EyQbEpbYOnGRwvjcDI1o8lBrIWzhDWQGyzkedYXELMofP2lB4ExeHehoSNddon3ETJUV45GJbYWrGDxE= Received: from VI1PR0502MB3056.eurprd05.prod.outlook.com (10.175.22.17) by AM4PR0501MB2753.eurprd05.prod.outlook.com (10.172.216.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1304.22; Tue, 1 Aug 2017 10:12:31 +0000 Received: from VI1PR0502MB3056.eurprd05.prod.outlook.com ([fe80::3da8:aae0:4a15:fbd8]) by VI1PR0502MB3056.eurprd05.prod.outlook.com ([fe80::3da8:aae0:4a15:fbd8%13]) with mapi id 15.01.1304.022; Tue, 1 Aug 2017 10:12:30 +0000 From: Matan Azrad To: Adrien Mazarguil CC: "dev@dpdk.org" , Thomas Monjalon , "Olga Shern" , "stable@dpdk.org" Thread-Topic: [PATCH] net/mlx4: workaround to verbs wrong error return Thread-Index: AQHTCgfEoeAeTg2ISkS844MN8M/g76JuAFFAgAFAkICAAAOmEA== Date: Tue, 1 Aug 2017 10:12:29 +0000 Message-ID: References: <1501499709-19873-1-git-send-email-matan@mellanox.com> <20170731141728.GO19852@6wind.com> <20170801094221.GQ19852@6wind.com> In-Reply-To: <20170801094221.GQ19852@6wind.com> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM4PR0501MB2753; 7:ZVpuNH0+99l8e+MNJLA377UF5lIP2PUrg8wNUFt4+VGCHFzjTzNgcvjUTDE4ntzv3jIA411d6ehrSEmdVs831hjVWxYfpqvo2mpfs8eQMagyuKQ/grq7rGZtEfk+1UDpSqkJfEWtsHgWx0kzg4CACKIHeiaimot9xZr/2GgDhbyW2VCZkJsivleFu8YdSSpshOdBM44X1rSt5incVFsvs4pAI9+QkpgylUT6/bsRo4K/emT5QGHcOuiY1Z+TKQL2ndjE9OsJlw5CCJyryT0ZlHPaDUpRCWKnNZzDuSfCgMv3uskvpDlBkeXLbm7OCK6MVSvZt8hgeMhNB5iscGc1XTXjdKY9e4TGHF4tr4UzbWWktPRDlZOaIYc6nPNKuYKYnfyFGHJvflziDnZ6qvk77tH6Mhnnikh5zPQCvwNRlC2pQNMu5YUr+Z2CO7DpnuOdA8B1VhK2JaxT+1uIRaIlwSedMbHneoSnyz/llDF+M2xbL+ptSDfWQx4VHyG5hGFqh+AJgCQ9g57xdI7wKe/nr6Qa+kW5fPDkoieniM3V90FbRFDnNfGLMyIZr8QVQlEW5h9NBP150hz4n7JnR+vwU+VZ3+L+xVy+2kKXu74Fe39cr4ulfLlKiOXfUVyjRHiPHqr8+jtFyuCgzALuGwdo1cQKCGApTMQKTAizc4mZfeV7MGLVkKhYPqgzqg8JLE4CSCfWKKZOMqjoXLW80yYnBhT8BqQXABdcCe0GuVUF4Z52eFPYkhtQ0xr4o5lRuqpt/4t0oeI0Vbo6DH5XpwHvg1lesmEzjY7iiXF6SxbN5N0= x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-ms-office365-filtering-correlation-id: 04e81441-7a9b-47af-37e8-08d4d8c5d19b x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(48565401081)(2017052603031)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:AM4PR0501MB2753; x-ms-traffictypediagnostic: AM4PR0501MB2753: x-exchange-antispam-report-test: UriScan:(788757137089); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(100000703101)(100105400095)(10201501046)(93006095)(93001095)(3002001)(6055026)(6041248)(20161123562025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123560025)(20161123564025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:AM4PR0501MB2753; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:AM4PR0501MB2753; x-forefront-prvs: 0386B406AA x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39400400002)(39410400002)(39860400002)(39850400002)(39450400003)(39840400002)(199003)(13464003)(377454003)(189002)(24454002)(81156014)(6436002)(9686003)(3280700002)(53936002)(3660700001)(54906002)(99286003)(55016002)(33656002)(5250100002)(6246003)(86362001)(6916009)(7736002)(229853002)(50986999)(76176999)(54356999)(110136004)(189998001)(2950100002)(2900100001)(101416001)(38730400002)(7696004)(97736004)(74316002)(305945005)(8676002)(3846002)(102836003)(478600001)(6506006)(68736007)(4326008)(93886004)(25786009)(8936002)(6116002)(66066001)(81166006)(2906002)(14454004)(53546010)(106356001)(105586002)(5660300001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM4PR0501MB2753; H:VI1PR0502MB3056.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Aug 2017 10:12:30.0506 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR0501MB2753 Subject: Re: [dpdk-dev] [PATCH] net/mlx4: workaround to verbs wrong error return X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Aug 2017 10:12:32 -0000 Hi Adrien > -----Original Message----- > From: Adrien Mazarguil [mailto:adrien.mazarguil@6wind.com] > Sent: Tuesday, August 1, 2017 12:42 PM > To: Matan Azrad > Cc: dev@dpdk.org; Thomas Monjalon ; Olga Shern > ; stable@dpdk.org > Subject: Re: [PATCH] net/mlx4: workaround to verbs wrong error return >=20 > Hi Matan, >=20 > On Mon, Jul 31, 2017 at 04:56:33PM +0000, Matan Azrad wrote: > > Hi Adrien > > > > > -----Original Message----- > > > From: Adrien Mazarguil [mailto:adrien.mazarguil@6wind.com] > > > Sent: Monday, July 31, 2017 5:17 PM > > > To: Matan Azrad > > > Cc: dev@dpdk.org; Thomas Monjalon ; Olga > Shern > > > ; stable@dpdk.org > > > Subject: Re: [PATCH] net/mlx4: workaround to verbs wrong error > > > return > > > > > > Hi Matan, > > > > > > On Mon, Jul 31, 2017 at 02:15:09PM +0300, Matan Azrad wrote: > > > > Current mlx4 OFED version has bug which returns error to ibv > > > > destroy functions when the device was plugged out, in spite of the > > > > resources were destroyed correctly. > > > > > > > > Hence, failsafe PMD was aborted, only in debug mode, when it tries > > > > to remove the device in plug-out process. > > > > > > > > The workaround removed the ibv destroy assertions. > > > > > > > > DPDK 18.02 release should work with OFED-4.2 which will include > > > > the verbs fix to this bug, then, this patch will be removed. > > > > > > > > Signed-off-by: Matan Azrad > > > > Cc: stable@dpdk.org > > > > > > Since this workaround is needed in order to validate hot-plug with > > > mlx4 compiled in debug mode due to a problem in Verbs, I don't think > > > stable@dpdk.org should be involved. > > > > > > > Ok I'll remove it. > > > > > What will be back-ported, once fixed, is the minimum OFED version to > > > install to properly benefit from hot-plug functionality. > > > > > > More comments about the patch below. > > > > > > > --- > > > > drivers/net/mlx4/mlx4.c | 70 > > > +++++++++++++++++++++++++++++++++++--------- > > > > drivers/net/mlx4/mlx4_flow.c | 22 ++++++++++---- > > > > 2 files changed, 73 insertions(+), 19 deletions(-) > > > > > > > > diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c > > > > index > > > > 8451f5b..94782c2 100644 > > > > --- a/drivers/net/mlx4/mlx4.c > > > > +++ b/drivers/net/mlx4/mlx4.c > > > > @@ -955,7 +955,10 @@ struct rxq * > > > > return 0; > > > > error: > > > > if (mr_linear !=3D NULL) > > > > - claim_zero(ibv_dereg_mr(mr_linear)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_dereg_mr(mr_linear); > > > > > > > > rte_free(elts_linear); > > > > rte_free(elts); > > > > @@ -992,7 +995,10 @@ struct rxq * > > > > txq->elts_linear =3D NULL; > > > > txq->mr_linear =3D NULL; > > > > if (mr_linear !=3D NULL) > > > > - claim_zero(ibv_dereg_mr(mr_linear)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_dereg_mr(mr_linear); > > > > > > > > rte_free(elts_linear); > > > > if (elts =3D=3D NULL) > > > > @@ -1052,9 +1058,15 @@ struct rxq * > > > > ¶ms)); > > > > } > > > > if (txq->qp !=3D NULL) > > > > - claim_zero(ibv_destroy_qp(txq->qp)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_qp(txq->qp); > > > > if (txq->cq !=3D NULL) > > > > - claim_zero(ibv_destroy_cq(txq->cq)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_cq(txq->cq); > > > > if (txq->rd !=3D NULL) { > > > > struct ibv_exp_destroy_res_domain_attr attr =3D { > > > > .comp_mask =3D 0, > > > > @@ -1070,7 +1082,10 @@ struct rxq * > > > > if (txq->mp2mr[i].mp =3D=3D NULL) > > > > break; > > > > assert(txq->mp2mr[i].mr !=3D NULL); > > > > - claim_zero(ibv_dereg_mr(txq->mp2mr[i].mr)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_dereg_mr(txq->mp2mr[i].mr); > > > > } > > > > memset(txq, 0, sizeof(*txq)); > > > > } > > > > @@ -1302,7 +1317,10 @@ static struct ibv_mr *mlx4_mp2mr(struct > > > > ibv_pd > > > *, struct rte_mempool *) > > > > DEBUG("%p: MR <-> MP table full, dropping oldest entry.", > > > > (void *)txq); > > > > --i; > > > > - claim_zero(ibv_dereg_mr(txq->mp2mr[0].mr)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_dereg_mr(txq->mp2mr[0].mr); > > > > memmove(&txq->mp2mr[0], &txq->mp2mr[1], > > > > (sizeof(txq->mp2mr) - sizeof(txq->mp2mr[0]))); > > > > } > > > > @@ -2355,7 +2373,10 @@ struct txq_mp2mr_mbuf_check_data { > > > > (void *)rxq, > > > > (*mac)[0], (*mac)[1], (*mac)[2], (*mac)[3], (*mac)[4], (*ma= c)[5], > > > > mac_index, priv->vlan_filter[vlan_index].id); > > > > - claim_zero(ibv_destroy_flow(rxq- > > > >mac_flow[mac_index][vlan_index])); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_flow(rxq->mac_flow[mac_index][vlan_index]); > > > > rxq->mac_flow[mac_index][vlan_index] =3D NULL; } > > > > > > > > @@ -2736,7 +2757,10 @@ struct txq_mp2mr_mbuf_check_data { > > > > DEBUG("%p: disabling allmulticast mode", (void *)rxq); > > > > if (rxq->allmulti_flow =3D=3D NULL) > > > > return; > > > > - claim_zero(ibv_destroy_flow(rxq->allmulti_flow)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_flow(rxq->allmulti_flow); > > > > rxq->allmulti_flow =3D NULL; > > > > DEBUG("%p: allmulticast mode disabled", (void *)rxq); } @@ > > > > -2796,7 > > > > +2820,10 @@ struct txq_mp2mr_mbuf_check_data { > > > > DEBUG("%p: disabling promiscuous mode", (void *)rxq); > > > > if (rxq->promisc_flow =3D=3D NULL) > > > > return; > > > > - claim_zero(ibv_destroy_flow(rxq->promisc_flow)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_flow(rxq->promisc_flow); > > > > rxq->promisc_flow =3D NULL; > > > > DEBUG("%p: promiscuous mode disabled", (void *)rxq); } @@ - > > > 2847,9 > > > > +2874,15 @@ struct txq_mp2mr_mbuf_check_data { > > > > rxq_mac_addrs_del(rxq); > > > > } > > > > if (rxq->qp !=3D NULL) > > > > - claim_zero(ibv_destroy_qp(rxq->qp)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_qp(rxq->qp); > > > > if (rxq->cq !=3D NULL) > > > > - claim_zero(ibv_destroy_cq(rxq->cq)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_cq(rxq->cq); > > > > if (rxq->channel !=3D NULL) > > > > claim_zero(ibv_destroy_comp_channel(rxq->channel)); > > > > if (rxq->rd !=3D NULL) { > > > > @@ -2864,7 +2897,10 @@ struct txq_mp2mr_mbuf_check_data { > > > > &attr)); > > > > } > > > > if (rxq->mr !=3D NULL) > > > > - claim_zero(ibv_dereg_mr(rxq->mr)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_dereg_mr(rxq->mr); > > > > memset(rxq, 0, sizeof(*rxq)); > > > > } > > > > > > > > @@ -4374,7 +4410,10 @@ struct txq_mp2mr_mbuf_check_data { > > > > priv_parent_list_cleanup(priv); > > > > if (priv->pd !=3D NULL) { > > > > assert(priv->ctx !=3D NULL); > > > > - claim_zero(ibv_dealloc_pd(priv->pd)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_dealloc_pd(priv->pd); > > > > claim_zero(ibv_close_device(priv->ctx)); > > > > } else > > > > assert(priv->ctx =3D=3D NULL); > > > > @@ -6389,7 +6428,10 @@ struct txq_mp2mr_mbuf_check_data { > > > > port_error: > > > > rte_free(priv); > > > > if (pd) > > > > - claim_zero(ibv_dealloc_pd(pd)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_dealloc_pd(pd); > > > > if (ctx) > > > > claim_zero(ibv_close_device(ctx)); > > > > if (eth_dev) > > > > diff --git a/drivers/net/mlx4/mlx4_flow.c > > > > b/drivers/net/mlx4/mlx4_flow.c index 925c89c..daa62e3 100644 > > > > --- a/drivers/net/mlx4/mlx4_flow.c > > > > +++ b/drivers/net/mlx4/mlx4_flow.c > > > > @@ -799,8 +799,11 @@ struct rte_flow_drop { > > > > struct rte_flow_drop *fdq =3D priv->flow_drop_queue; > > > > > > > > priv->flow_drop_queue =3D NULL; > > > > - claim_zero(ibv_destroy_qp(fdq->qp)); > > > > - claim_zero(ibv_destroy_cq(fdq->cq)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_qp(fdq->qp); > > > > + ibv_destroy_cq(fdq->cq); > > > > rte_free(fdq); > > > > } > > > > } > > > > @@ -860,7 +863,10 @@ struct rte_flow_drop { > > > > priv->flow_drop_queue =3D fdq; > > > > return 0; > > > > err_create_qp: > > > > - claim_zero(ibv_destroy_cq(cq)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_cq(cq); > > > > err_create_cq: > > > > rte_free(fdq); > > > > err: > > > > @@ -1200,7 +1206,10 @@ struct rte_flow * > > > > (void)priv; > > > > LIST_REMOVE(flow, next); > > > > if (flow->ibv_flow) > > > > - claim_zero(ibv_destroy_flow(flow->ibv_flow)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_flow(flow->ibv_flow); > > > > rte_free(flow->ibv_attr); > > > > DEBUG("Flow destroyed %p", (void *)flow); > > > > rte_free(flow); > > > > @@ -1278,7 +1287,10 @@ struct rte_flow * > > > > for (flow =3D LIST_FIRST(&priv->flows); > > > > flow; > > > > flow =3D LIST_NEXT(flow, next)) { > > > > - claim_zero(ibv_destroy_flow(flow->ibv_flow)); > > > > + /* Current verbs does not allow to check real > > > > + * errors when the device was plugged out. > > > > + */ > > > > + ibv_destroy_flow(flow->ibv_flow); > > > > flow->ibv_flow =3D NULL; > > > > DEBUG("Flow %p removed", (void *)flow); > > > > } > > > > -- > > > > 1.8.3.1 > > > > > > > > > > This approach looks way too intrusive. How about making the > > > claim_zero() definition not fail but still complain when compiled > > > against a broken Verbs version instead? > > > > > > #include "mlx4_autoconf.h" > > > > > > [...] > > > > > > #ifndef HAVE_BROKEN_VERBS > > > #define claim_zero(...) assert((__VA_ARGS__) =3D=3D 0) #else /* > > > HAVE_BROKEN_VERBS */ #define claim_zero(...) \ > > > (void)(((__VA_ARGS__) =3D=3D 0) || \ > > > DEBUG("Assertion `" # __VA_ARGS__ "' failed (IGNORED)")) > > > #endif /* HAVE_BROKEN_VERBS */ > > > > > > You could use auto-config-h.sh to generate the HAVE_BROKEN_VERBS > > > definition in mlx4_autoconf.h (see mlx4 Makefile) based on some > > > symbol, macro or type that only exists or doesn't exist yet in > > > problematic releases for instance. > > > > > > > I agree with the dependence on broken verbs but there are other places > > in mlx4 code which use claim_zero assertion, So this suggestion will > > hurt other validations. >=20 > Well, half broken is no better than completely broken in my opinion, so w= hile > Verbs is being repaired, users debugging the mlx4 PMD will temporarily ge= t > debug traces without the ensuing abort(). At least the behavior will be > consistent. >=20 > Think about it, they already have to go out of their way to enable > CONFIG_RTE_LIBRTE_MLX4_DEBUG, if they know they aren't using hot-plug > but still use a buggy Verbs version, they can disable HAVE_BROKEN_VERBS t= o > revert to the normal behavior. >=20 priv_flow_validate and priv_mac_addr_add functions calls also are wrapped b= y claim_zero, These are not ibv_destroy functions and don't depend only in broken verbs, The user want to be aborted in those cases otherwise he would have put ther= e trace print=20 as you suggest. > > What's about to create new define depend on broken verbs for the specif= ic > assertions? > > It will be still intrusive but more accurate. >=20 > One reason I prefer the code to remain unchanged is that I'm currently > refactoring the entire PMD. Maintaining the above patch (picking the righ= t > ibv_*() calls that return a consistent value) will be difficult and an in= trusive > patch won't be reverted easily once Verbs is fixed. > You can just find all claim_zero_new and replace it with claim_zero.=20 =20 > All these claim_zero() checks ensure the PMD destroys Verbs resources in > the proper order (e.g. a flow before the QP it is associated with). If th= e > return value of any of these cannot be relied on, it's useless to only ch= eck > some of them. priv_flow_validate and priv_mac_addr_add functions are not destroy verbs re= sources. >=20 > Moreover if ibv_destroy_something() wrongly returns an error when the > device is unplugged, I think this can happen to the calls not part of you= r > patch, i.e. all of them, so working around it at the macro definition lev= el > makes sense. I checked with failsafe tests and found that only the specific destroy func= tions are problematic. >=20 > If you don't know what symbol can be relied on in OFED 4.2 to define > HAVE_BROKEN_VERBS (which is just an example, you can use another name > BTW), maybe you can add a compilation option to enable manually in case o= f > trouble? Something verbose like: >=20 > CONFIG_RTE_LIBRTE_MLX4_DEBUG_BROKEN_VERBS_ASSERT=3Dn >=20 > Which will have to be documented. >=20 > -- > Adrien Mazarguil > 6WIND