From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 308A6A06CB; Sun, 13 Nov 2022 19:31:26 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1ACF140150; Sun, 13 Nov 2022 19:31:25 +0100 (CET) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140049.outbound.protection.outlook.com [40.107.14.49]) by mails.dpdk.org (Postfix) with ESMTP id 660664014F for ; Sun, 13 Nov 2022 19:31:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=URh9jNNsrG1pwvDBnAot1V2S7TboZNMctUgloiM+u8o=; b=6B9nYa/Wn3FH6VzyqsRXhGiptHBcFZ4prnMgTIKSFHezEAUpBmO9YfHBskKs7SsFDV1rwpaoMDrAYWqvXf2GhgD13VhuUcHcNPlJeBKWtN3F83GUoAc/AJL2pYs+kVm9DmHIL101AGbce9yqplsanQnxWAKtoWsW5F475N9BiBo= Received: from FR3P281CA0068.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:4b::15) by DB9PR08MB8650.eurprd08.prod.outlook.com (2603:10a6:10:3d2::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5834.6; Sun, 13 Nov 2022 18:31:21 +0000 Received: from VI1EUR03FT063.eop-EUR03.prod.protection.outlook.com (2603:10a6:d10:4b:cafe::2a) by FR3P281CA0068.outlook.office365.com (2603:10a6:d10:4b::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5813.17 via Frontend Transport; Sun, 13 Nov 2022 18:31:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VI1EUR03FT063.mail.protection.outlook.com (100.127.144.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5813.12 via Frontend Transport; Sun, 13 Nov 2022 18:31:20 +0000 Received: ("Tessian outbound f394866f3f2b:v130"); Sun, 13 Nov 2022 18:31:20 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: e7409443adbf68f4 X-CR-MTA-TID: 64aa7808 Received: from a1548ab82ae7.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 58AAD7DC-C0F8-452F-BE10-89D7D1EAD601.1; Sun, 13 Nov 2022 18:31:10 +0000 Received: from EUR05-AM6-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id a1548ab82ae7.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Sun, 13 Nov 2022 18:31:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ex4q3Cb/nqKMLkEZwwoFw0IP5nGzorFeulXG+znPznfagQCbxyAl3sFbeJmtuzUfbRyRsQinb5V2ZPDi30bc+XZvir7ZK/VX3nBeFkREH9L9p7mpjlUKpfmtXYoEOPJ6VGIlK3mOKtkTdSFQeOfAXLA6XkxTM4TU/aB5stOUBm/jQUUGS8qzzieZjbJ+8N2i/H0Ck2Q7KAivXsNOi1CPnRzjacojpIYyUa6fQE9937dTCm8ONw6USJnQ6sKkhfpll9eUnDCgvC8qRCrM9p86crnVde9902TIuJp+Er78dXvzIX1hyJRz+1MctJeXpmTnug/K18k1uCVuQ2amrBIqOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=URh9jNNsrG1pwvDBnAot1V2S7TboZNMctUgloiM+u8o=; b=ebwgY+Ira53v2RKRKk7pHZgg3eWoG4BDMIW4XwNQ7JVG2HW+U5n3Sj1ElloCcW9Nm8ooTQySUUP5CjlDPVPRWRJqoUkXpv+lsZlDmb6WjQD81XthpQt/z62Id6/yQlQ/e31lCPgVOIWx7lL3Dzq8PFihtDJ8J3BJMBY+SNlmu2Vc8+bvdPgB5dji9wDqzdNI+IhxPyKk9B+5cWuOm/ASOeEFtdP9F1gJwTSmFKT1OEnjcfFFPuWksXciwEeAHrJqYOD8yPXq2RAKVK3shqE3zp8+uth/u7FHsa4WyrPRLAo1p3hpI2gG2PQyr2rxgyjBR2I1sWWzXHIebHeeqAdiTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=URh9jNNsrG1pwvDBnAot1V2S7TboZNMctUgloiM+u8o=; b=6B9nYa/Wn3FH6VzyqsRXhGiptHBcFZ4prnMgTIKSFHezEAUpBmO9YfHBskKs7SsFDV1rwpaoMDrAYWqvXf2GhgD13VhuUcHcNPlJeBKWtN3F83GUoAc/AJL2pYs+kVm9DmHIL101AGbce9yqplsanQnxWAKtoWsW5F475N9BiBo= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by DBBPR08MB6073.eurprd08.prod.outlook.com (2603:10a6:10:1f7::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5834.6; Sun, 13 Nov 2022 18:31:07 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::3fb6:b7b2:1e8d:11d6]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::3fb6:b7b2:1e8d:11d6%8]) with mapi id 15.20.5834.006; Sun, 13 Nov 2022 18:31:07 +0000 From: Honnappa Nagarahalli To: =?iso-8859-1?Q?Morten_Br=F8rup?= , "dev@dpdk.org" , "olivier.matz@6wind.com" , "andrew.rybchenko@oktetlabs.ru" CC: Kamalakshitha Aligeri , nd , nd Subject: RE: [RFC]: mempool: zero-copy cache get bulk Thread-Topic: [RFC]: mempool: zero-copy cache get bulk Thread-Index: AdjxGTKWl0t76eJiQb2E+CV6pXxvaQAlLH4gAQ2jalA= Date: Sun, 13 Nov 2022 18:31:07 +0000 Message-ID: References: <98CBD80474FA8B44BF855DF32C47DC35D8748B@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D8748B@smartserver.smartshare.dk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 1EB635BC43DE1548AC99D404D5A97CDD.0 Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: DBAPR08MB5814:EE_|DBBPR08MB6073:EE_|VI1EUR03FT063:EE_|DB9PR08MB8650:EE_ X-MS-Office365-Filtering-Correlation-Id: 9dd4cc41-60c8-4abd-8a73-08dac5a54307 x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: LQv66Ep/3z+tfj3fl1e4R6ARM1Cp58WZyqFbQvlyAkxc/npGjoPfCidm9cKhrPPwfhmkOHP9X2+RYm5wrxx06zMBKy6z2xm+iqrTqfGx7ALX1+nQnBeZfyzaxIDPmBZBoBLVg3Q7r8DcxgXBa7UQEqIJWoSDJvoekU1IS0J7CDKGCqaMdifVmT1cz/Zomw5IYtFLx55cfl9KRww9+gBU84RCB6aLOeOuQY55yH/kG9YnlB7eUNHOMAN9B0mHkrIrnSuhk3rkS+HZaFBz+D5VRMbvLvD7Wtn5pa06d9zPuGl4ssJeZZGJUlatpPGQLGq9rVd/rlndaA2tnxKuTFkPR7oOF0FeyasekjIcqf6WMoT4Udenx5QRkysRJV0SD0NR707OzdJAshFN5Tg6LNfhKBQgTrdIwiSHHj+MiUS4PIjm6YmkMvQUwBXQPGrmA9vE64fvoWLshndLznL8LbHHHvzzI9d0K6IK5i7uxMLLUEK+Z3Z/8VkK41huG2wAEWZsiuiJyanZZFLd+IQCx+5Pma+oL+vnFwmhplLtdlqFhPn1ojjB/slTrB6ZPVww6z7yo41c71lXbVOwSRGDFqQFun1mJ/Pnjd3KxB2iQ5IqQbeEsAnGMbq2feLZ3WxSvIZmCmnOpzUY4DQMHUn6p+HrtYYVNBnOJK2r5HEUYP9Uq2lji6sMQADQu77emF/X16gQ5CBL1lbiAm6YBQr0cUBsIbyG+G5k/3rtsD+S8yuKEeATo815tGeF/WgHiIB9FhzX41ZERkSXq7pyXUIqw1EYPje+xjIYbJF+1gJo5jp13BE= X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(346002)(376002)(396003)(366004)(136003)(39850400004)(451199015)(26005)(33656002)(7696005)(6506007)(9686003)(478600001)(55016003)(86362001)(966005)(110136005)(54906003)(316002)(186003)(66574015)(52536014)(8936002)(38070700005)(2906002)(76116006)(66946007)(66556008)(66476007)(122000001)(38100700002)(64756008)(66446008)(41300700001)(83380400001)(5660300002)(66899015)(71200400001)(8676002)(4326008); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB6073 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VI1EUR03FT063.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 90629252-aec8-47d8-4bfb-08dac5a53acc X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oT3Q3XEBu9Oe7DNlY5Wi6VtvMbnG7CnydaoK10OG78Udj6K1qBYlnX8r4vRBO0V/X1gUmzzU5bfMvdqBBbPSpBKWJJ7iTWYbA+1YqVoDSoLS5016hWM8hhZ+ggHi2bEraBwigEpvnJgUqfORSZNYOy6L+mVmnp5Ifby5CRubwORxH8GbglY57MLxhz5XgCczrYbEdrNK+ou4nlPdugnCIAWmI1MuGkXHDiVMI8ksze1HXOLH3pInfc0aLutgt4C5Q/Lg2Qoo3WKIti/3+ZBG/a3T3ifyOn6eTUOGEoGDygtbiSXPZLLdXUb3seCjlGvejdLu3C7gDkUmMdF4blEo0FlQ2U2JlCVvCwC/Fqa2nYSU69VvLj36cFNP9dlL6kpqAU5treYXFKos0LBLBJSQNOw4Wf3jqRb3o6+Pmzqb7X3QMRKk2vN2wbHUM0hIP+UiQOMC4WXZUpwvCK34T+3IoFabYS89/G2Ll7b7NkRQZ/h/GYOKBT2ZCnq4okIGT6bzGreD5E712++79cc+RlvmzUeqQAoHOsx1+TO7/u3SfJHpmHG6b24vVn2E9C81P0FkG0uBkukEfjtcG+/yfE7XaBN2xFINrY/25J8ZEVlJOdWI87IMox6zdEML1VA7JI/zvsnXQrB6gK8pTHSwIWAdz4CJYugOIg29UfGJAzx98TJerEZVGJ6gvUUQtWPeLeyGdeUiB9qHciVF/2S0J+2zIJUqzoFSjleubeUzctswFtsCrspa9zNfNZA5ZdSU79utVN/YFI/xfuYqGki8rVn8DnaDk6VLueNoQdNAPowJmNLn5WNu/MEE0H+Z6s1G5CI7 X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230022)(4636009)(39850400004)(346002)(396003)(376002)(136003)(451199015)(36840700001)(46966006)(36860700001)(2906002)(70206006)(70586007)(5660300002)(83380400001)(41300700001)(8936002)(4326008)(52536014)(8676002)(86362001)(55016003)(33656002)(40480700001)(82740400003)(81166007)(82310400005)(356005)(54906003)(110136005)(478600001)(966005)(7696005)(9686003)(47076005)(66574015)(6506007)(26005)(66899015)(336012)(186003)(316002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Nov 2022 18:31:20.9783 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9dd4cc41-60c8-4abd-8a73-08dac5a54307 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: VI1EUR03FT063.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB8650 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org >=20 > > From: Morten Br=F8rup > > Sent: Saturday, 5 November 2022 14.19 > > > > Zero-copy access to the mempool cache is beneficial for PMD > > performance, and must be provided by the mempool library to fix [Bug > > 1052] without a performance regression. > > > > [Bug 1052]: https://bugs.dpdk.org/show_bug.cgi?id=3D1052 > > > > > > This RFC offers two conceptual variants of zero-copy get: > > 1. A simple version. Few comments inline for this. > > 2. A version where existing (hot) objects in the cache are moved to > > the top of the cache before new objects from the backend driver are > > pulled in. I think there is no compelling use case for this. We could come up with the= oretical use cases, but may be we should avoid complicating the code till s= omeone hits the problem. In the run to completion model, if this situation occurs, the configuration= of the cache can be changed to avoid this. In the pipeline model, we will not hit this condition as the refill thresho= ld can be a multiple of cache size. > > > > I would like some early feedback. Also, which variant do you prefer? > > > > Notes: > > * Allowing the 'cache' parameter to be NULL, and getting it from the > > mempool instead, was inspired by rte_mempool_cache_flush(). >=20 > "instead" -> "in this case" >=20 > > * Asserting that the 'mp' parameter is not NULL is not done by other > > functions, so I omitted it here too. > > > > NB: Please ignore formatting. Also, this code has not even been > > compile tested. >=20 > And I just spotted an error: the rte_memcpy() length field must be multip= lied > by sizeof(void*). >=20 > > > > 1. Simple version: > > > > /** > > * Get objects from a mempool via zero-copy access to a user-owned > > mempool cache. > > * > > * @param cache > > * A pointer to the mempool cache. > > * @param mp > > * A pointer to the mempool. > > * @param n > > * The number of objects to prefetch into the mempool cache. > > * @return > > * The pointer to the objects in the mempool cache. > > * NULL on error > > * with rte_errno set appropriately. > > */ > > static __rte_always_inline void * > > rte_mempool_cache_get_bulk(struct rte_mempool_cache *cache, > > struct rte_mempool *mp, > > unsigned int n) > > { > > unsigned int len; > > > > if (cache =3D=3D NULL) > > cache =3D rte_mempool_default_cache(mp, rte_lcore_id()); I do not think we should support this. > > if (cache =3D=3D NULL) { > > rte_errno =3D EINVAL; > > goto fail; > > } This could be under debug flags. This is a data plane function, we should a= void additional checks. > > > > rte_mempool_trace_cache_get_bulk(cache, mp, n); > > > > len =3D cache->len; > > > > if (unlikely(n > len)) { > > unsigned int size; > > > > if (unlikely(n > RTE_MEMPOOL_CACHE_MAX_SIZE)) { > > rte_errno =3D EINVAL; > > goto fail; > > } Same here, this being a data plane function, we should avoid these checks. = This checks for something that is fundamentally incorrect in the applicatio= n. > > > > /* Fill the cache from the backend; fetch size + requested - > > len objects. */ > > size =3D cache->size; > > > > ret =3D rte_mempool_ops_dequeue_bulk(mp, &cache->objs[len], siz= e > > + n - len); > > if (unlikely(ret < 0)) { > > /* > > * We are buffer constrained. > > * Do not fill the cache, just satisfy the request. > > */ > > ret =3D rte_mempool_ops_dequeue_bulk(mp, &cache->objs[len], > > n > > - len); > > if (unlikely(ret < 0)) { > > rte_errno =3D -ret; > > goto fail; > > } > > > > len =3D 0; > > } else > > len =3D size; > > } else > > len -=3D n; > > > > cache->len =3D len; > > > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1); > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n); > > > > return &cache->objs[len]; > > > > fail: > > > > RTE_MEMPOOL_STAT_ADD(mp, get_fail_bulk, 1); > > RTE_MEMPOOL_STAT_ADD(mp, get_fail_objs, n); > > > > return NULL; > > } > > > > > > 2. Advanced version: > > > > /** > > * Get objects from a mempool via zero-copy access to a user-owned > > mempool cache. > > * > > * @param cache > > * A pointer to the mempool cache. > > * @param mp > > * A pointer to the mempool. > > * @param n > > * The number of objects to prefetch into the mempool cache. > > * @return > > * The pointer to the objects in the mempool cache. > > * NULL on error > > * with rte_errno set appropriately. > > */ > > static __rte_always_inline void * > > rte_mempool_cache_get_bulk(struct rte_mempool_cache *cache, > > struct rte_mempool *mp, > > unsigned int n) > > { > > unsigned int len; > > > > if (cache =3D=3D NULL) > > cache =3D rte_mempool_default_cache(mp, rte_lcore_id()); > > if (cache =3D=3D NULL) { > > rte_errno =3D EINVAL; > > goto fail; > > } > > > > rte_mempool_trace_cache_get_bulk(cache, mp, n); > > > > len =3D cache->len; > > > > if (unlikely(n > len)) { > > unsigned int size; > > > > if (unlikely(n > RTE_MEMPOOL_CACHE_MAX_SIZE)) { > > rte_errno =3D EINVAL; > > goto fail; > > } > > > > /* Fill the cache from the backend; fetch size + requested - > > len objects. */ > > size =3D cache->size; > > > > if (likely(size + n >=3D 2 * len)) { > > /* > > * No overlap when copying (dst >=3D len): size + n - len >= =3D > > len. > > * Move (i.e. copy) the existing objects in the cache to > > the > > * coming top of the cache, to make room for new objects > > below. > > */ > > rte_memcpy(&cache->objs[size + n - len], &cache->objs[0], > > len); >=20 > Length is bytes, not number of objects, so that should be: >=20 > rte_memcpy(&cache->objs[size + n - len], &cache->objs[0], len * > sizeof(void*)); >=20 > > > > /* Fill the cache below the existing objects in the cache. > > */ > > ret =3D rte_mempool_ops_dequeue_bulk(mp, &cache->objs[0], > > size + n - len); > > if (unlikely(ret < 0)) { > > goto constrained; > > } else > > len =3D size; > > } else { > > /* Fill the cache on top of any objects in it. */ > > ret =3D rte_mempool_ops_dequeue_bulk(mp, &cache->objs[len], > > size + n - len); > > if (unlikely(ret < 0)) { > > > > constrained: > > /* > > * We are buffer constrained. > > * Do not fill the cache, just satisfy the request. > > */ > > ret =3D rte_mempool_ops_dequeue_bulk(mp, &cache- > > >objs[len], n - len); > > if (unlikely(ret < 0)) { > > rte_errno =3D -ret; > > goto fail; > > } > > > > len =3D 0; > > } else > > len =3D size; > > } > > } else > > len -=3D n; > > > > cache->len =3D len; > > > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1); > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n); > > > > return &cache->objs[len]; > > > > fail: > > > > RTE_MEMPOOL_STAT_ADD(mp, get_fail_bulk, 1); > > RTE_MEMPOOL_STAT_ADD(mp, get_fail_objs, n); > > > > return NULL; > > } > > > > > > Med venlig hilsen / Kind regards, > > -Morten Br=F8rup